1. INDIGO-DataCloud Software What/Where/How
Doina Cristina Duma
INFN-CNAF

2. Outline INDIGO – DC Software – what
INDIGO – DC artefacts repositories – where
INDIGO – DC installation & configuration - how
INDIGO-DataCloud

3. INDIGO-DataCloud

4. WHY
INDIGO-DataCloud

5. ElectricIndigo
INDIGO-DataCloud

6. INDIGO – DC Software

7. What – INDIGO-DC Service Catalog
INDIGO-DataCloud delivers open source software components tailored to scientific communities and to e-infrastructures, aimed to increase ease of use and effectiveness in the exploitation of Cloud resources
addresses the challenge of developing open source software,
deployable in the form of a data/computing platform,
aimed to scientific communities
structural biology, earth sciences, physics, bioinformatics,
cultural heritage, astrophysics, life sciences and climatology.
designed to be deployed on public or private Clouds
integrated with existing resources or e-infrastructures.
Scientific
INDIGO-DataCloud

8. What – Data Center Solutions
Fairshare Scheduler for OS &ON
Partition Director Service for Batch and Cloud resources
CPR, IM, OCCI support for OpenStack and OpenNebula
Extended OpenStack and OpenNebula Functionalities - OPIE, nova-docker, Keystone AAI Support, TOSCA in HEAT, ONEDock, DockerHub Sync, OpenStack Client & Nova Client, OpenStack Identity Authentication Library
Network Orchestration Wrapper
INDIGO-DataCloud

9. What – Data Solutions Global Data Access – Onedata
Storage Quality of Service and Data Lifecycle support - CDMI
INDIGO-DataCloud

10. What – Automated Solutions
Scientific
INDIGO-DataCloud

11. What – High-level User Oriented Sol.
Userspace Container Support
Data Mining and Analytics for eScience Server – ophidia
Future Gateways (Programmable Scientific Portal)
INDIGO Plug-ins for scientific workflow systems
INDIGO Mobile Toolkit
New! Orchent, the orchestrator client
INDIGO-DataCloud

12. What – Common Solutions
IAM, IDH, TTS
INDIGO-DataCloud

13. What – INDIGO-DC Service Catalog/Download - https://www
Scientific
INDIGO-DataCloud

14. INDIGO – DC artefacts repositories

15. INDIGO-DC Software Development Flow
T3.1 Software quality assurance
T3.2 Software release and maintenance
T3.4
Exploitation
WP4
WP5
WP6
External
Service
Providers
Development
infrastructure
Integration
infrastructure
Preview
infrastructure
T3.3 Pilot services
software deployment
software use
software delivery
WP2
Application
Use-cases
Users
Production
work 15

16. From Heterogeneity
Heterogeneity in INDIGO DataCloud development context
Nature of products
new developments, enhancements to existing products, upstream contributions
Scope
Diversity of middleware stacks (OpenStack, OpenNebula, CMD)
Diversity of target environments (data intensive, HTC, HPC, Cloud, Grid)
Definition
Multiple programming languages and standards

17. … to the implementation
software improvement cycle and the integration / release / software quality process
Software Quality assurance
Code style checking
Unit testing
Functional/Integration testing
Code Review
Documentation
Software Release and Maintenance
Release management
Publish release schedules
Manage project public repositories
Continuous integration
Continuous delivery
Software maintenance & support
Change Management processes
Support to released software
Problem Management process

18. Development and Integration infrastructure
CESNET
rOCCI
PSNC
Indigokepler
indigo-omt
LIP/INCD
OpenNebula: ONEDock
Nova-Docker
FutureGateway
UPV IM
CLUES
TOSCA
IFCA/CSIC
OOI
OPIE
INFN Bari
Kubernetes
Mesos
Chronos
DESY
dCache
CERN
Magnun
CNAF/INFN
IAM
Oneprovider
KIT
CDMI-QoS
TTS
Cyfronet
Onedata

19. Demos are performed in the preview testbed
Pilot Preview testbed
CNAF/INFN
IAM
OneData
CDMI-QoS
Orchestrator
CloudProviderRanker
Zabbix-wrapper
SLAManager
CMDB
LIP/INCD
OOI
IAM connector
Nova-Docker
OS Identity Authentication library
ONEDock
rOCCI server
TTS
Java-syncrepos
Cloud-info-provider IM
FG API server
FG Portal
LiferayIAM
Indigo Kepler
Ophidia
DESY
dCache
IFCA/SIC
ooi
nova-docker
java-syncrepos
UPV
INFN-Padova
Synergy
INFN-Bari
Kubernetes:
Marathon
Chronos
Mesos
KIT
CDMI-QOS
Demos are performed in the preview testbed

20. Release and Maintenance process

21. Release Timeline INDIGO-1 Full updates Standard updates
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
INDIGO-1
Full updates
Standard updates
Security updates
INDIGO-2
Release Date
End of Full Updates
End of Standard Updates
End of Security Updates & EOL
INDIGO-1 MidnightBlue
08/08/2016
31/01/2017
31/03/2017
31/05/2017
INDIGO-2 EletricIndigo
14/04/2017
30/09/2017
30/11/2017
31/01/2018

22. INDIGO-DC Artefacts Artefacts
Binary packages, source packages, tarballs (binaries & sources)
O.S.:
CentOS 7
Ubuntu 14.04, (LTS)
CMF:
OpenStack: Liberty, Newton
OPENNebula: 4.14, 5.2
Container images for applications and appliances
Signing - guarantee the provenance
Plus documentation
Release notes
Major & minor releases

23. Repositories

24. Repositories

25. Repositories – Other Resources

26. Repositories – Other Resources

27. Documentation - GitBook

28. Repositories – Other Resources

29. Configuration

30. Repositories – Other Resources

31. Code Repositories - GitHub

32. Repositories – Other Resources

33. Continuous Integration - Jenkins

34. Repositories – Other Resources

35. Other Repos

36. Repositories – Other Resources

37. Repos config files

38. INDIGO – DC Install. & Config.

39. Summary
Installing the Operating Systems and Cloud Management Frameworks
Operating Systems
Cloud Management Frameworks
Enable the INDIGO - DataCloud packages repositories
Giving INDIGO - DataCloud repositories precedence over EPEL
Enable the INDIGO - DataCloud Containers repositories
Important note on automatic updates

40. Operating Systems CentOS 7: Ubuntu 16.04: https://www.centos.org/
Mirrors:
Docker Containers:
Ubuntu 16.04:
, Ubuntu Install Guide
Docker Containers:

41. Cloud Management Frameworks
OpenStack Newton
Please follow the official OpenStack Newton Installation Guides:
for RedHat & CentOS
Please enable the use of Newton RDO repository by using:
for Ubuntu 16.04
Please enable the use of Newton CloudArchive by using:
OpenNebula 4.14 & 5.2.X
Please follow the official OpenNebula 4.14 Installation Guide for CentOS & Ubuntu or OpenNebula 5.X Installation Guide for CentOS & Ubuntu
# sudo yum install -y
# sudo add-apt-repository cloud-archive:newton

42. Enable indigo-dc packages repos
The packages repositories have the following structure:
INDIGO-DC production (stable): indigo/{1,2}/<platform>/<basearch>/{base|updates}
stable and signed, well tested software components, recommended to be installed on production-sites
Third-party: indigo/{1,2}/<platform>/<basearch>/third-party
packages that are not part of INDIGO, or not part of the base OS or EPEL, but used as dependencies by other INDIGO components
INDIGO-DC testing: indigo-testing/{1,2}/<platform>/<basearch>
packages that will become part of the next stable distribution; in the certification and validation phase.
INDIGO-DC preview: indigo-preview/{1,2}/<platform>/<basearch>
S igned packages that will become part of the next stable update, available for technical-previews
where
<basearch> is currently: x86_64, SRPMS, tgz
<platform> is currently: centos7, ubuntu

43. Enable indigo-dc packages repos
All packages are signed with the INDIGO - DataCloud gpg key. The public key can be downloaded from here, and the fingerprint from here. Please import the key BEFORE starting!
for CentOS7 save the key under /etc/pki/rpm-gpg/
for Ubuntu:
# rpm --import
# wget -q -O - | sudo apt-key add -

44. “Enforce” indigo-dc repos
It is strongly recommended that INDIGO repositories take precedence over EPEL when installing and upgrading packages. For manual configuration:
you must install the yum-priorities** plugin and ensure that its configuration file, /etc/yum/pluginconf.d/priorities.conf is as follows:
[ main ] enabled = 1 check_obsoletes = 1

45. INDIGO-DC repos – automatic config
INDIGO-2 production repositories are available at:
YUM & APT configuration files are available at:
CentOS7 -
Ubuntu
Install INDIGO - DataCloud repositories :
CentOS7:
Ubuntu 16.04:
# wget # yum localinstall -y indigodc-release el7.centos.noarch.rpm
# wget # dpkg -i indigodc-release_ _amd64.deb

46. “Enable” indigo-dc containers repos
On the DockerHub Registry, INDIGO - DataCloud has organized the repositories under two Organizations:
indigodatacloud, for Core Services
indigodatacloudapps, for Applications
Containers present in those repositories and released in INDIGO-2 are tagged with "indigo_2" tag and signed, leveraging the Docker’s trust features so that users can pull trusted images.
Currently, content trust is disabled by default. You must enable it by setting the DOCKER_CONTENT_TRUST environment variable, like bellow:
For more details regarding the "Content Trust in Docker" please read Docker's Documentation
Content trust is associated with the TAG portion of an image.
See next slide for ex. of a correct use of INDIGO - DataCloud images
export DOCKER_CONTENT_TRUST=1

47. Container Use – Core Services

48. Container Use – Applications

49. Releases – MidnightBlue & ElectricIndigo

50. First release Release version Indigo-1 Release name MidnightBlue
Release date
8th August 2016
Release repository
Release documentation
Software license
Apache 2.0
Supported Operating Systems
Linux (CentOS 7 and Ubuntu 14.04)
Supported cloud frameworks
OpenStack Liberty and OpenNebula 4.14
Packages
208 RPMs, DEBs and tarballs
40 Docker containers
Release announcements

51. Second release Release version Indigo-2 Release name ElectricIndigo
Release date
14th April 2017
Release repository
Release documentation
Software license
Apache 2.0
Supported Operating Systems
Linux (CentOS 7 and Ubuntu 16.04)
Supported cloud frameworks
OpenStack Newton and OpenNebula
Packages
413 RPMs, DEBs and tarballs
40 Docker containers
Release announcements

52. https://www.indigo-datacloud.eu Better Software for Better Science.
Thank you
Better Software for Better Science.
October 2016
Patrick Fuhrmann - The INDIGO-DataCloud MidnightBlue Release