Smart Cities and Communities and Social Innovation Bando MIUR D.D. 391/Ric. del 5 luglio 2012 Cloud Environment Automation: from infrastructure deployment to application monitoring Cristina Aiftimiei, Alessandro Costantini, Diego Michelotto et al. INFN-CNAF, IT {cristina.aiftimiei, alessandro.costantini}@cnaf.infn.it

Open City Platform (OCP) Industrial research project Funded by the Italian Ministry of University and Research (MIUR) started in 2014 Project intends to research, develop and test new technology solutions open, interoperable and usable on-demand on the Cloud, innovative organizational models sustainable over time delivery of services from Local Government and Regional Administrations to Citizens and Companies

OCP main changelles Scientific and technological challenges Design and reengineering Cloud applications Federated identity management and its trust relationship Open data and the Open Service and integration into business models Disaster recovery as a Service Legal, organizational, functional challenges and new business Define new organizational models and public governance where regions have the role of infrastructure intermediaries Adherence to new models regulations

OCP Architecture The OCP components are aimed to create a complete integration between the different Cloud stacks:

OCP Architecture OpenStack and Ubuntu open source products The OCP components are aimed to create a complete integration between the different Cloud stacks: • The orchestrator module, able to implement business logic and it provides support for the automation of process workflows and complex services composition based on the orchestration of simple services with reference to specific activities.

OCP Architecture The OCP components are aimed to create a complete integration between the different Cloud stacks: • Identity management module, that represents an authentication framework for the management of users digital identities. This consists of giving the ability to control who has access to the network, what it can do, what services can actually use.

OCP Architecture The OCP components are aimed to create a complete integration between the different Cloud stacks: • Open Data e Open Service Engine devoted to provide a fully support to the generation and use of data to provide Open Data and Open Service from homogeneous data sources.

OCP Architecture The OCP components are aimed to create a complete integration between the different Cloud stacks: • Marketplace for Service/Citizen components addressed to users representing customers that are looking for reusable services and components the fist; making available a place where services can be searched or offered to the territory the latest.

OCP Architecture The OCP components are aimed to create a complete integration between the different Cloud stacks: • Infrastructure as a Service (IaaS) platform based on OpenStack, suitably configured by capitalizing on the pioneering experiences made in the INFN Cloud-infrastructure, and other national and european projects.

IaaS installation scenarios Manual installation and configuration (“hardcore”) Pros better understanding of OpenStack dependencies between components more control over configurations Cons requires basic knowledge of Linux OS, bash and network configuration error-prone and time-consuming Full-automated installation via Fuel Easy installation through the graphical interface Enables subsequent changes and new deployments Initial configuration cannot be changed Custom configuration difficult to be applied.

Cloud Environment Automation A semi-automatic installation method Designed to take the advantages of the methods presented More control over configurations Easy installation through the graphical interface Flexible to meet the architectural requirements of Data Centers Leverage on automation tools Foreman and Puppet

Automation tools Puppet Foreman Framework open source for the management and configuration of ICT systems Descrive server configurations using a declarative language Maintain over time a specific configuration (state) for a specific service/server Foreman Framework for the lifecycle management of virtual and physical server Rapid deployment of services and applications Easy automation of repetitive actions Proactive Management of servers

Cloud Environment Automation Architecture Master Node – Configuration Management & Monitoring CM Foreman Puppet DNS Zabbix Monitoring Node01,02,03 – Services (RHMK) HAProxy Percona/MySQL MongoDB Keepalived RabbitMQ Zookeeper Node07,08– Controller Node09,10– Network Node11,12,N – Compute IDENTITY Keystone NETWORKING COMPUTE NETWORKING Neutron agents DASHBOARD Horizon An overview of the infrastructure where all the main components and services are defined. Hosts virtual or physical -Master Node the configuration management services are hosted (Foreman and Puppet) can eventually host a monitoring server Zabbix -RHMK (RabbitMQ, Haproxy, MongoDB, Keepalived) services services can be hosted in different nodes if the High Availability is required services are external to the OpenStack distribution and they provide: (i) databases for the OpenStack services (Percona/Mysql and MongoDB) (ii) a messaging system to let the OpenStack services to connect and scale (RabbitMq) (iii) Load Balancing (Haproxy, Zookeeper, Keepalived). A point to stress out here is that the whole RHMK services have been identified as a Role in Puppet, while each mentioned service is a Profile. -Storage service Ceph version Hammer as a block and object storage -Controller contains the common OpenStack services which are defined as Puppet Profiles. The configuration variables are hosted in the Puppet Role named Controller. The service is designed to run in High Availability but it can be deployed as a single service also -Network: it contains the Neutron agent services which are defined as Puppet Profiles and configured separately. The configuration variables are hosted in the Puppet Role called Network. The service is designed to run in High Availability but it can be deployed as a single service also. The Network service can be joint to the Controller service if imposed by architectural needs. -Compute contains the Compute services which are defined as Puppet Profiles. The configuration variables are hosted in the Puppet Role called Compute that permits to deploy a compute node and add it to the OpenStack infrastructure at any time. Nova compute Neutron OVS agents IMAGE COMPUTE Nova Glance Ceilometer TELEMETRY Node04,05,06 – CEPH NETWORKING Neutron server Distributed FS BLOCK STORAGE CEPH Cinder

Network Architecture OCP Network currently support the configuration of five different networks as from the OpenStack documentation (i) Public network, used to provide internet access to nodes and services (ii) Management network, used for internal communication between OpenStack Components and RHMK services (iii) Data network, used for VM data communication within the cloud deployment (iv) Storage Network, used to provide storage access from OpenStack components and tenants (v) External network, used to provide VMs with Internet access in some deployment scenarios. As an added feature, multiple External networks can be configured and deployed by setting appropriate variables in the Network Puppet Role.

Conclusion & Future Work Cloud Environment Automation A new semi-automatic method for IaaS installation and configuration Developed in the OCP industrial research project Address the different requirements and realities Flexible IaaS configuration and management High Avaialbility and Network management support Ceph adopted as block and object storage backend Fine grained variable configuration Graphical User Interface support

Conclusion & Future Work New Cloud Environment Automation features Full support to the OpenStack Identity API v3 Automatic upgrade of the OCP-IaaS layer to a new OpenStack version Automatic methods an tools for the installation and configuration of the PaaS layer CloudFormation as a Service