1. Platform & Engineering Services CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Agile Infrastructure Project Overview : Status and How to use it Vítor Gouveia, vitor.gouveia@cern.ch IT-PES-PS

2. Platform & Engineering Services Agenda Agile Infrastructure (AI) Overview Current Status of the AI project Schedule of the AI project How to use the AI : –Creation and configuration of VMs Current support Questions ? 2/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

3. Platform & Engineering Services Agile Infrastructure (AI) Overview CERN IT is changing strategy for machine provision and configuration. Cloud Infrastructure as a Service (IaaS) –virtual delivery of computing resources in the form of hardware, networking, and storage services. It may also include the delivery of operating systems and virtualization technology to manage the resources. The goal of the Agile Infrastructure Project is to: –Provide and efficient and flexible infrastructure service which enables new applications and supports existing ones while exploiting increased capacity and multi-site facilities within the current man power resources 3/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

4. Platform & Engineering Services Agile Infrastructure (AI) Overview 4/19 User documentation being written at: –http://information-technology.web.cern.ch/book/cern- puppet-development-user-guidehttp://information-technology.web.cern.ch/book/cern- puppet-development-user-guide Internal documentation about the AI project: –https://twiki.cern.ch/twiki/bin/view/AgileInfrastructure/WebHomehttps://twiki.cern.ch/twiki/bin/view/AgileInfrastructure/WebHome Agile Infrastructure Components: –Configuration: Puppet, Foreman, Hiera –Provision: OpenStack –Other Services GIT, Koji AgendaOverviewStatusScheduleDemo Reports problems Questions

5. Platform & Engineering Services Agile Infrastructure (AI) Overview Agile Infrastructure Configuration Components: –Puppet Open source configuration management tool like Quattor Declarative language to describe system configuration Puppet programs are called “manifests” –Conceptually like a cdb template The core of the Puppet language is the resource declaration file { 'testfile': path => '/tmp/testfile', ensure => present, mode => 0640, content => "I'm a test file.", } A singleton collections of resources in puppet is called class Tutorial about puppet: http://docs.puppetlabs.com/learning/http://docs.puppetlabs.com/learning/ CERN puppet user guide (work in progress): http://information-technology.web.cern.ch/book/cern-puppet- development-user-guide 5/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

6. Platform & Engineering Services Agile Infrastructure Configuration Components (Cont.): –Puppet ( Cont.) Facter –Puppet tools which examines our system and normalizes all the information about it »creates a set of variables and passes them off to puppet –Source of information about the machines (CPUs, RAM, network etc.) –Functionally similar to CDB hardware/network profiles, but provides information dynamically from various data sources –Hiera Data store used by puppet hierarchical-based lookup system 6/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

7. Platform & Engineering Services Agile Infrastructure (AI) Overview Agile Infrastructure Configuration Components (Cont.): –Foreman Current web front-end for Puppet and OpenStack Groups hosts (node) into hostgroups of similar configuration Hostgroup –Essentially the same a cdb cluster –In a hostgroup several features are aggregated together –Ex: Cluster of a LAMP server ( should include the puppet classes responsible for apache, mysql and php) –Can be hierarchical Ex: 1 st level ~= cdb cluster, 2 nd /3 rd level ~= cdb customization Real time information about hosts status (node) Rebuild machines from scratch High-level machine configuration (~= CDB profile) –Possible to configure a hostgroup or just a single host https://judy.cern.ch/users/login 7/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

8. Platform & Engineering Services Agile Infrastructure Provision Component: –OpenStack (Hamster) Software in use for the CERN private cloud. –Principal organizational structure Tenants –Resource container for a project/VO/… Tenant users can freely create volumes, VMs etc. within a certain quota of total Disk, Cores, RAM, etc… https://hamster.cern.ch/dashboard 8/19

9. Platform & Engineering Services Agile Infrastructure (AI) Overview Other Services –GIT Distributed version control system GIT in the AI project: –GIT branches map to Foreman environments ( ~= cdb stages ) –Inside a branch it is possible to find manifests and modules Modules: Collection of configuration components Ex: apache, afs, ssh, sssd Similar to cdb services and cdb components Can use other modules Stored in repository root in modules/ 9/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

10. Platform & Engineering Services Other Services (Cont.) –GIT ( Cont.) Manifests: They are specific for each hostgroup Should not be reusable May reference several modules Conceptually similar to cdb cluster/customization templates Describe which modules/puppet classes/puppet resources should be applied to a set of machines (hostgroup) Ex: lxplus, lxBatch Stored in repository root in manifest/ https://gitgw.cern.ch/gitweb/ 10/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

11. Platform & Engineering Services Other Services (Cont.) –KOJI Creates RPMs and publish to YUM repositories Instead of the swrepsrv 11/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

12. Platform & Engineering Services Current Status of the AI project Pre-Production Service running: –OpenStack Essex, Foreman 1.0 and puppet 2.7 Several IT services running on the AI –EX: SLC6 Batch machines Deploy puppet configured VMs on OpenStack –Configuration and Management through web portal (Foreman) –VM creation still requires using command-line scripts for the time being 12/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

13. Platform & Engineering Services Schedule of the AI project Deploy into production: –End of January 2013 Folsom (Openstack) suitable for implementing the 'Cattle‘ –machines that may die without impact on production Ex: batch machines –Not suitable yet for ‘pet’ machines Even less if High-availability is necessary –Ex: VOBOX –Eventually Openstack will be integrated in Foreman (together with Puppet) It will be possible to create and manage VMs through Foreman Only one website! –Performance and stability will improve over time 13/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

14. Platform & Engineering Services How to use the AI From now on you can start using the AI: –Attention: Everything is in pre-production –Things will change!! What are you going to see? –How to create a VM in the AI –How to configure it More info: https://twiki.cern.ch/twiki/bin/view/AgileInfrastructure/VOCGettingStarted 14/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

15. Platform & Engineering Services How to use the AI Mailings lists and egroups to join: –lxvoadm-Authorized-Users Gives Access to [aiadm02.cern.ch | aiadm03.cern.ch] server Aiadm is for AI what lxvoadm is for Quattor –puppet-users CERN puppet discussion –ai-vobox-migration-discussion: Gives you access to create VMs in the AI project Best way to provide feedback 15/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

16. Platform & Engineering Services 16/19

17. Platform & Engineering Services Current support This is not a production service so official support lines don't exist yet. All Feedback about the VOBox migration to AI should be reported through the mailing list ai-vobox-migration-discussion 17/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

18. Platform & Engineering Services Current support Upcoming meetings… –Topics to be covered GIT, branches Partitions + resize Monitoring + custom sensors Koji + RPMs Accounting etc… –Starting from January every two weeks… Thursdays afternoon 18/19 AgendaOverviewStatusScheduleDemo Reports problems Questions

19. Platform & Engineering Services Questions??? 19/19