INDIGO-DataCloud Software What/Where/How Doina Cristina Duma INFN-CNAF cristina.aiftimiei@cnaf.infn.it

Outline INDIGO – DC Software – what INDIGO – DC artefacts repositories – where INDIGO – DC installation & configuration - how INDIGO-DataCloud

INDIGO-DataCloud

WHY INDIGO-DataCloud

ElectricIndigo INDIGO-DataCloud

INDIGO – DC Software

What – INDIGO-DC Service Catalog INDIGO-DataCloud delivers open source software components tailored to scientific communities and to e-infrastructures, aimed to increase ease of use and effectiveness in the exploitation of Cloud resources addresses the challenge of developing open source software, deployable in the form of a data/computing platform, aimed to scientific communities structural biology, earth sciences, physics, bioinformatics, cultural heritage, astrophysics, life sciences and climatology. designed to be deployed on public or private Clouds integrated with existing resources or e-infrastructures. https://www.indigo-datacloud.eu/software Scientific INDIGO-DataCloud

What – Data Center Solutions Fairshare Scheduler for OS &ON Partition Director Service for Batch and Cloud resources CPR, IM, OCCI support for OpenStack and OpenNebula Extended OpenStack and OpenNebula Functionalities - OPIE, nova-docker, Keystone AAI Support, TOSCA in HEAT, ONEDock, DockerHub Sync, OpenStack Client & Nova Client, OpenStack Identity Authentication Library Network Orchestration Wrapper INDIGO-DataCloud

What – Data Solutions Global Data Access – Onedata Storage Quality of Service and Data Lifecycle support - CDMI INDIGO-DataCloud

What – Automated Solutions Scientific INDIGO-DataCloud

What – High-level User Oriented Sol. Userspace Container Support Data Mining and Analytics for eScience Server – ophidia Future Gateways (Programmable Scientific Portal) INDIGO Plug-ins for scientific workflow systems INDIGO Mobile Toolkit New! Orchent, the orchestrator client INDIGO-DataCloud

What – Common Solutions IAM, IDH, TTS INDIGO-DataCloud

What – INDIGO-DC Service Catalog/Download - https://www Scientific INDIGO-DataCloud

INDIGO – DC artefacts repositories

INDIGO-DC Software Development Flow T3.1 Software quality assurance T3.2 Software release and maintenance T3.4 Exploitation WP4 WP5 WP6 External Service Providers Development infrastructure Integration infrastructure Preview infrastructure T3.3 Pilot services software deployment software use software delivery WP2 Application Use-cases Users Production work 15

From Heterogeneity Heterogeneity in INDIGO DataCloud development context Nature of products new developments, enhancements to existing products, upstream contributions Scope Diversity of middleware stacks (OpenStack, OpenNebula, CMD) Diversity of target environments (data intensive, HTC, HPC, Cloud, Grid) Definition Multiple programming languages and standards

… to the implementation software improvement cycle and the integration / release / software quality process Software Quality assurance Code style checking Unit testing Functional/Integration testing Code Review Documentation Software Release and Maintenance Release management Publish release schedules Manage project public repositories Continuous integration Continuous delivery Software maintenance & support Change Management processes Support to released software Problem Management process

Development and Integration infrastructure CESNET rOCCI PSNC Indigokepler indigo-omt LIP/INCD OpenNebula: ONEDock Nova-Docker FutureGateway UPV IM CLUES TOSCA IFCA/CSIC OOI OPIE INFN Bari Kubernetes Mesos Chronos DESY dCache CERN Magnun CNAF/INFN IAM Oneprovider KIT CDMI-QoS TTS Cyfronet Onedata

Demos are performed in the preview testbed Pilot Preview testbed CNAF/INFN IAM OneData CDMI-QoS Orchestrator CloudProviderRanker Zabbix-wrapper SLAManager CMDB LIP/INCD OOI IAM connector Nova-Docker OS Identity Authentication library ONEDock rOCCI server TTS Java-syncrepos Cloud-info-provider IM FG API server FG Portal LiferayIAM Indigo Kepler Ophidia DESY dCache IFCA/SIC ooi nova-docker java-syncrepos UPV INFN-Padova Synergy INFN-Bari Kubernetes: Marathon Chronos Mesos KIT CDMI-QOS Demos are performed in the preview testbed

Release and Maintenance process

Release Timeline INDIGO-1 Full updates Standard updates Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul INDIGO-1 Full updates Standard updates Security updates INDIGO-2 Release Date End of Full Updates End of Standard Updates End of Security Updates & EOL INDIGO-1 MidnightBlue 08/08/2016 31/01/2017 31/03/2017 31/05/2017 INDIGO-2 EletricIndigo 14/04/2017 30/09/2017 30/11/2017 31/01/2018

INDIGO-DC Artefacts Artefacts Binary packages, source packages, tarballs (binaries & sources) O.S.: CentOS 7 Ubuntu 14.04, 16.04 (LTS) CMF: OpenStack: Liberty, Newton OPENNebula: 4.14, 5.2 Container images for applications and appliances Signing - guarantee the provenance Plus documentation Release notes Major & minor releases

Repositories

Repositories

Repositories – Other Resources

Repositories – Other Resources

Documentation - GitBook

Repositories – Other Resources

Configuration

Repositories – Other Resources

Code Repositories - GitHub

Repositories – Other Resources

Continuous Integration - Jenkins

Repositories – Other Resources

Other Repos

Repositories – Other Resources

Repos config files

INDIGO – DC Install. & Config.

Summary Installing the Operating Systems and Cloud Management Frameworks Operating Systems Cloud Management Frameworks Enable the INDIGO - DataCloud packages repositories Giving INDIGO - DataCloud repositories precedence over EPEL Enable the INDIGO - DataCloud Containers repositories Important note on automatic updates

Operating Systems CentOS 7: Ubuntu 16.04: https://www.centos.org/ Mirrors: http://mirror.centos.org/centos/7/ Docker Containers: https://hub.docker.com/_/centos/ Ubuntu 16.04: http://www.ubuntu.com/ http://releases.ubuntu.com/xenial/ , Ubuntu Install Guide Docker Containers: https://hub.docker.com/_/ubuntu/

Cloud Management Frameworks OpenStack Newton Please follow the official OpenStack Newton Installation Guides: for RedHat & CentOS Please enable the use of Newton RDO repository by using: for Ubuntu 16.04 Please enable the use of Newton CloudArchive by using: OpenNebula 4.14 & 5.2.X Please follow the official OpenNebula 4.14 Installation Guide for CentOS & Ubuntu or OpenNebula 5.X Installation Guide for CentOS & Ubuntu # sudo yum install -y https://repos.fedorapeople.org/openstack/openstack-newton/rdo-release-newton-4.noarch.rpm # sudo add-apt-repository cloud-archive:newton

Enable indigo-dc packages repos The packages repositories have the following structure: INDIGO-DC production (stable): indigo/{1,2}/<platform>/<basearch>/{base|updates} stable and signed, well tested software components, recommended to be installed on production-sites Third-party: indigo/{1,2}/<platform>/<basearch>/third-party packages that are not part of INDIGO, or not part of the base OS or EPEL, but used as dependencies by other INDIGO components INDIGO-DC testing: indigo-testing/{1,2}/<platform>/<basearch> packages that will become part of the next stable distribution; in the certification and validation phase. INDIGO-DC preview: indigo-preview/{1,2}/<platform>/<basearch> S igned packages that will become part of the next stable update, available for technical-previews where <basearch> is currently: x86_64, SRPMS, tgz <platform> is currently: centos7, ubuntu

Enable indigo-dc packages repos All packages are signed with the INDIGO - DataCloud gpg key. The public key can be downloaded from here, and the fingerprint from here. Please import the key BEFORE starting! for CentOS7 save the key under /etc/pki/rpm-gpg/ for Ubuntu: # rpm --import http://repo.indigo-datacloud.eu/repository/RPM-GPG-KEY-indigodc # wget -q -O - http://repo.indigo-datacloud.eu/repository/RPM-GPG-KEY-indigodc | sudo apt-key add -

“Enforce” indigo-dc repos It is strongly recommended that INDIGO repositories take precedence over EPEL when installing and upgrading packages. For manual configuration: you must install the yum-priorities** plugin and ensure that its configuration file, /etc/yum/pluginconf.d/priorities.conf is as follows: [ main ] enabled = 1 check_obsoletes = 1

INDIGO-DC repos – automatic config INDIGO-2 production repositories are available at: http://repo.indigo-datacloud.eu/repository/indigo/2/ YUM & APT configuration files are available at: CentOS7 - http://repo.indigo-datacloud.eu/repos/2/indigo2.repo Ubuntu 16.04 - http://repo.indigo-datacloud.eu/repos/2/indigo2-ubuntu16_04.list Install INDIGO - DataCloud repositories : CentOS7: Ubuntu 16.04: # wget http://repo.indigo-datacloud.eu/repository/indigo/2/centos7/x86_64/base/indigodc-release-2.0.0-1.el7.centos.noarch.rpm # yum localinstall -y indigodc-release-2.0.0-1.el7.centos.noarch.rpm # wget http://repo.indigo-datacloud.eu/repository/indigo/2/ubuntu/dists/xenial/main/binary-amd64/indigodc-release_2.0.0-1_amd64.deb # dpkg -i indigodc-release_2.0.0-1_amd64.deb

“Enable” indigo-dc containers repos On the DockerHub Registry, INDIGO - DataCloud has organized the repositories under two Organizations: indigodatacloud, for Core Services indigodatacloudapps, for Applications Containers present in those repositories and released in INDIGO-2 are tagged with "indigo_2" tag and signed, leveraging the Docker’s trust features so that users can pull trusted images. Currently, content trust is disabled by default. You must enable it by setting the DOCKER_CONTENT_TRUST environment variable, like bellow: For more details regarding the "Content Trust in Docker" please read Docker's Documentation Content trust is associated with the TAG portion of an image. See next slide for ex. of a correct use of INDIGO - DataCloud images export DOCKER_CONTENT_TRUST=1

Container Use – Core Services

Container Use – Applications

Releases – MidnightBlue & ElectricIndigo

First release Release version Indigo-1 Release name MidnightBlue Release date 8th August 2016 Release repository http://repo.indigo-datacloud.eu Release documentation https://www.gitbook.com/book/indigo-dc/indigo-datacloud-releases/details Software license Apache 2.0 Supported Operating Systems Linux (CentOS 7 and Ubuntu 14.04) Supported cloud frameworks OpenStack Liberty and OpenNebula 4.14 Packages 208 RPMs, DEBs and tarballs 40 Docker containers Release announcements http://repo.indigo-datacloud.eu/INDIGONews.rss.xml e-mail: indigo-announce@lists.indigo-datacloud.eu

Second release Release version Indigo-2 Release name ElectricIndigo Release date 14th April 2017 Release repository http://repo.indigo-datacloud.eu Release documentation https://www.gitbook.com/book/indigo-dc/indigo-datacloud-releases/details Software license Apache 2.0 Supported Operating Systems Linux (CentOS 7 and Ubuntu 16.04) Supported cloud frameworks OpenStack Newton and OpenNebula 5.2170 Packages 413 RPMs, DEBs and tarballs 40 Docker containers Release announcements http://repo.indigo-datacloud.eu/INDIGONews.rss.xml e-mail: indigo-announce@lists.indigo-datacloud.eu

https://www.indigo-datacloud.eu Better Software for Better Science. Thank you https://www.indigo-datacloud.eu Better Software for Better Science. October 2016 Patrick Fuhrmann - The INDIGO-DataCloud MidnightBlue Release