1. Software Management and Pilot Services
Better Software for Better Science.
Jorge Gomes, LIP
WP3 manager and WP4 member
INDIGO-DataCloud Final Review
Brussels, 16-17/11/2017
RIA

2. WP3 - Software Management and Pilot Services
WP Overview
Objectives and tasks
Activities
Task T3.1 Software quality assurance (lead: IFCA)
Task T3.2 Software release and maintenance (lead: INFN)
Task T3.3 Pilot services (lead: LIP)
Task T3.4 Exploitation (lead: EGI)
Summary
Metrics and KPIs
Final remarks
Thanks to:
Pablo Orviz
Cristina Aiftimiei
Mario David
Matthew Viljoen
Ludek Matyska
and all WP3 members
Support the software lifecycle, from development to deployment and exploitation.
INDIGO-DataCloud Final Review
Software Management and Pilot Services

3. Software Management and Pilot Services
WP3 Objectives
Tasks
Project Objectives
T3.1
Software quality assurance
Define and monitor the quality assurance process for the developed software
i) code style, ii) unit testing, iii) functional and integration testing, iv) documentation, v) code review, vi) automated deployment
Collect metrics and Key Performance Indicators (KPIs)
Alignment with QA procedures of external Open Source projects to ease upstream contributions
Promote and adopt DevOps practices
Continuous integration & Continuous delivery
T3.2
Software release and maintenance
Software Maintenance & Support
Coordination of corrective, adaptive, perfective and preventive maintenance activities, defining
Software Maintenance, Problem & Change Management processes
Provisioning of support for released software
Release Management
define release plans & schedules, enforce the agreed software release policies to the software services and components
Coordinate the implementation of Continuous Integration & Delivery processes
Manage the software artefacts & documentation repositories
T3.3
Pilot services
Provide infrastructures and services needed for the integration and testing of software:
Services to support the development, Software Quality Assurance, release management, maintenance and support.
Test infrastructure to support software development.
Integration infrastructure to support the integration between components and services.
Pilot preview infrastructure with the components and services released by INDIGO-DataCloud.
T3.4
Exploitation
Make the software releases available in the relevant e-infrastructures, fostering adoption & enlarging the user base,
Staged rollout of the software in production environments at several providers.
Establish and coordinate a Service Providers Board (SPB).
Make the software discoverable (e.g. targeting UMD and EGI applications database).
Exploitation strategy including a services portfolio, and complement user oriented exploitation.
INDIGO-DataCloud Final Review
Software Management and Pilot Services

4. Activities and Achievements
INDIGO-DataCloud Final Review
Software Management and Pilot Services

5. Task T3.1 Software quality assurance
INDIGO-DataCloud Final Review
Software Management and Pilot Services

6. T3.1: SQA Achievements in INDIGO-2 (I)
Unit testing coverage => increasing trend
Total average value is 63.34% (70% recommended in D3.1)
~60% of products over the 70% threshold
~80% of products over 50% coverage
Low coverage products: several are external contributions (different SQA baseline)
INDIGO-DataCloud Final Review
Software Management and Pilot Services

7. T3.1: SQA Achievements in INDIGO-2 (II)
Automated deployment
Number of products delivering automated deployment solutions has increased
Products not fulfilling: mainly clients or libraries => no configuration involved
Ansible is mostly used
INDIGO-DataCloud Final Review
Software Management and Pilot Services

8. T3.1: SQA Achievements in INDIGO-2 (III)
Documentation
All the product’s documentation is now available in markdown format, stored in GitHub repositories
Treated as code, benefits:
Portable: can be trivially moved elsewhere
(Code)-reviewed: typos/wrong modifications spotted easily before merging in production
Automated rendering: once in production, documentation is automatically rendered as books, online available (GitBook)
Code style
100% of the products aligned with community or de-facto standards
Functional testing
Automated or manual reports provided for each release/update
Code review
Performed through GitHub’s pull-requests
INDIGO-DataCloud Final Review
Software Management and Pilot Services

9. T3.1: DevOps adoption (automation) (I)
Products/middleware
Delivery is now automated => Continuous Delivery (CD)
Step forward from Continuous Integration (CI) scenario in INDIGO-1
Source code is not only validated but 1) automatically packaged and 2) uploaded to the appropriate means of distribution (RPMs/DEBs, Docker images, tarballs)

10. T3.1: DevOps adoption (automation) (II)
User communities
User applications’ development and distribution => profit from DevOps practices
Expertise gathered from INDIGO core services development and delivery
The schema is similar to the Continuous Delivery scenario described:
Application’s source code is checked => code style (CI)
Docker image is built and published to indigodatacloudapps repository (CD)
Application deployment => Ansible role
(NEW) Application validation
The application is automatically instantiated using the brand new Docker image generated in the previous step
Application is executed and the results validated with the expected output
Prototype applied to two applications => DisVis and PowerFit
Perfect case studies since they build their own applications, not relying on external applications=> access to codebase
INDIGO-DataCloud Final Review
Software Management and Pilot Services

11. T3.1: DevOps adoption (automation) (III)
DevOps workflow for DisVis app development
DevOps pipeline (automated steps):
Source code validation
Python’s PEP8
Packaging
Docker image building using Ansible roles
Delivering
Upload Docker image to indigodatacloudapps DockerHub repository
Application validation
Based on fixed inputs and expected results, the application is executed and the results validated
Pipeline execution time: <7 minutes
INDIGO-DataCloud Final Review
Software Management and Pilot Services

12. Task T3.2 Software release and maintenance
INDIGO-DataCloud Final Review
Software Management and Pilot Services

13. Software Lifecycle Management
”make the certified software components available as a set of coherent high quality releases, supported by an efficient maintenance process”
By:
Defining processes..
Planning ..
Managing … the results/artefacts
INDIGO-DataCloud Final Review
Software Management and Pilot Services

14. Lifecycle Management - Journey
D3.3 P2
Submitted
Final Results
D3.2 P2
Submitted
Updated Plan
D3.1 P1
Approved
Initial Plan

15. INDIGO-DC Timeline
End of project
(*) - timeline after the end of project - commitments collected from the PTs & respective partners
(**) - lightweight INDIGO-DC SOFTWARE STACK PROVIDERS COLLABORATION AGREEMENT being established.

16. INDIGO-1 - Release

17. INDIGO-2 - Release

18. INDIGO-DC - Maintenance (I)
14 Updates providing
174 component releases

19. INDIGO-DC - Maintenance (II)
Corrective Maintenance
Priority driven -> GitHub Dashboard
Adaptive Maintenance
Porting to new platforms:
CentOS7, Ubuntu & 16.04
Openstack Liberty & Newton
OpenNebula 4.14 & 5.2
Perfective Maintenance
Improve software functionality according to users’ needs
Preventive Maintenance
Code styles checking & Introduction of regression testing
INDIGO-DataCloud Final Review
Software Management and Pilot Services

20. INDIGO-DC - Maintenance (III)
INDIGO-DataCloud Final Review
Software Management and Pilot Services

21. INDIGO-DC - Maintenance (IV)
65 EMT phone-conf over 30 months
INDIGO-DataCloud Final Review
Software Management and Pilot Services

22. Activities after INDIGO-DC
Activity
During INDIGO-DC
After INDIGO-DC
INDIGO-DC products
Support & Maintenance
INDIGO-DC PTs
PTs are providing continued support and maintenance to user communities
via their SU & new EU projects
Continuous Integration & Delivery
Jenkins
PTs internal tools &
Standard tools of upstream distributions (like OpenStack) (*)
Maintenance coordination
EMT
New EU projects - in particular XDC & DEEP-H for dev. & bug fixes, EOSC-Hub for maintenance (**)
Release coordination
New EU projects
Support coordination
Direct relationships and SLAs between PTs and user communities
Support can continue to be provided through GGUS & GitHub Issues
Repositories
INDIGO-DC
repositories
Upstream repositories & Docker Hub
INDIGO-DC repository (maintained active), EGI CMD and UMD repos
(*) - new projects agreed to continue support of CI & CD using & sharing the same tool (Jenkins) & infrastructure (provided by INDIGO).
(**) - commitments and future projects supporting development and/or maintenance.
INDIGO-DataCloud Final Review
Software Management and Pilot Services

23. Software Lifecycle Management
2 Major Releases
14 Updates
65 EMT phone-confs
196 source code repositories
Packages:
CentOS7, Ubuntu14.04, 16.04
OpenStack Liberty & Newton
OpenNebula 4,14 & 5.2
~ 730 registered issues (98 open, 633 closed)
GGUS:- 7 tickets
INDIGO-DataCloud Final Review
Software Management and Pilot Services

24. Software Management and Pilot Services
Task T3.3 Pilot services
INDIGO-DataCloud Final Review
Software Management and Pilot Services

25. T3.3: Status of Pilot preview testbed I
Consolidation of resources into the pilot preview testbed:
Several endpoints used for development are now part of the pilot preview.
Increase the number of resources in the testbed for larger scale testing:
IaaS level: 5 Openstack production sites, including T-Systems plus 2 Kubernetes, Marathon clusters.
Several CDMI-QoS endpoints with diverse underlying storage technologies:
HPSS, dCache, S3, StoRM, NFS
The testbed remains operational after the end of project:
Enabling development, testing and demonstration.
Basis for new projects.
INDIGO-DataCloud Final Review
Software Management and Pilot Services

26. T3.3: Pilot preview map # VMs (or hosts) 520 # VCPUs 2460
RAM (GB) 6300
Storage (TB) 100
# Public IPs 90
T3.3: Pilot preview map
DESY
CDMI server
dCache
Oneprovider
T-Systems
Openstack -> PaaS
Kubernetes
Oneprovider
INFN-Padova
IAM connector
Synergy
OOI
nova-docker
KIT
CDMI server
Oneprovider
PSNC
CDMI server
IFCA/SIC
ooi
IAM connector
Java-syncrepos
Oneprovider
Cloud InfoProvider
CNAF/INFN
IAM
Oneprovider
CDMI server
Orchestrator
CloudProviderRanker
Zabbix-wrapper
SLAManager
CMDB
Partition Director (DynPart)
LIP/INCD
OOI
IAM connector
nova-docker
ONEDock
rOCCI server
NOW
ONE-FaSS
WaTTS
Java-syncrepos
Cloud-info-provider IM
Onezone + Onaprovider
FG API server
FG Portal
LiferayIAM
Indigo Kepler
Ophidia
CMCC
Ophidia web server
INFN-Bari
IAM connector
CDMI server
Oneprovider
Kubernetes:
Marathon
Chronos
Mesos
UPV IM
CLUES

27. T3.3: Pilot preview https://ophidialab.cmcc.it:11732/
indigo-ui.ncg.ingrid.pt
infra01.ncg.ingrid.pt

28. T3.3: Inclusion of public cloud providers
Several public cloud providers have been integrated into the Pilot Preview infrastructure:
T-Systems Open Telekom Cloud, Amazon AWS and Microsoft Azure.
Used as part of the overall INDIGO-DataCloud platform services, at the IaaS level and Federated Data Storage/Management.
Used to perform stress and scalability tests by the use cases.
Examples of success:
IM (Infrastructure Manager) and Orchestrator:
Automated instantiation of stacks of Virtual Machines.
Instantiation of Docker orchestration platforms, such as Mesos clusters (through the use of Tosca templates).
Onedata:
T-Systems Open Telekom Cloud: Integration of a Onedata Oneprovider service instance into Pilot Preview infrastructure.
Storage spaces for WP2 users have been granted to support several use cases
INDIGO-DataCloud Final Review
Software Management and Pilot Services

29. T3.3: Elasticity and stress tests
Galaxy cluster use case
Scalability tests performed on deploying Mesos clusters at several IaaS sites, including public cloud providers
Automatic stack deployment of Galaxy portals with elastic clusters, or HTCondor clusters.
Clusters of ~100 Virtual Machines;
Configured, contextualised, all necessary services in less than one hour.
~100 small sized clusters of about 10 VMs and 100 cores
Deployed on Openstack IaaS sites.
4 distinct Virtual HTCondor cluster continuously running.
Open Telekom Cloud:
The largest cluster with > 700 worker nodes
clusters: > 180 VMs,> 1400 cores, ~ 2.7TB of RAM.
Using maximum capacity available at the provider
No bottleneck found
DODAS use case CERN/CMS
INDIGO-DataCloud Final Review
Software Management and Pilot Services

30. T3.x Security
Architecture and design
Steering and mgmt of releases
Security incidents
coordination
Technical
Board
Engineering
Management
Team
Security
Group
Project bodies
Orthogonal to the project activities and across all stages
SQA
- follow code styles
- tests: unit, regression,
functional
- auto config and install
SW RELEASES
- security updates
- signed artifacts
SW MAINTENANCE
- vulnerability handling
INFRASTRUCTURE
- development
- pilot preview
- commercial providers
- testing by use cases
- use of TCS certificates
- bridging to EGI CSIRT
EXPLOITATION
- stage rollout
- external infrastructures
- gathering feedback
ASSESSMENT
- evaluation of tools
- assessment of
selected products
using SWAMP
- assessment of live
Onedata deployment
Project activities
Services and Infrastructure resources
INDIGO-DataCloud Final Review
Software Management and Pilot Services

31. Software Management and Pilot Services
Task T3.4 Exploitation
INDIGO-DataCloud Final Review
Software Management and Pilot Services

32. T3.4: Staged rollout and Early Adoption
INDIGO Products went through a Staged Rollout and Early Adopter process (See D3.2 and D3.3)
Some new components in INDIGO-2
All of the products have received reports from Early Adopters
INDIGO products have been deployed on multiple sites
Stage Rollout
Pilot Preview and development testbed
Other external parties
INDIGO Products mapped to existing EGI distribution channels
CMD, UMD, EGI DockerHub, AppDB
Inclusion is ongoing and status is summarized in D3.3
INDIGO-DataCloud Final Review
Software Management and Pilot Services

33. T3.4: Staged Rollout workflow

34. T3.4: Exploitation (user communities)
Use case / User community
Algæbloom (Lifewatch) 1
Medical Imaging Biobanks (EuroBioImaging) 2
Molecular Dynamics Simulations (INSTRUCT) 3a
Astronomical Data Archive (LBT) 3b
Archive System for the Cherenkov Telescope Array (CTA) 4
Disvis, Powerfit, Haddock (WeNMR) 5
Climate models inter comparison data analysis (ENES) 6
eCulture Science Gateway/R (eCulture) 8
Galaxy as a Cloud service (ELIXIR-ITA) 9
MOIST-multidisciplinary oceanic information system (EMSO) 10
Data repository platform (DARIAH) 11
Container-based cluster on demand for data analysis (LHC-CMS)
Component
User community P#
Onedata
0, 1, 2, 3a, 3b, 6, 8, 9, 11
IAM
0, 1, 2, 3a, 8, 10, 11
Orchestrator (TOSCA/Mesos&Chronos)
0, 1, 8, 10, 11
FutureGateway
0, 2, 5, 8, 9
Infrastructure Manager
1, 8, 11
udocker, ONEDock, CLUES, docker containers, Ophidia, OMT, Orchent, TTS/WaTTS, TOSCA parser
1, 2, 3a, 4, 5, 8, 9, 11
INDIGO-DataCloud Final Review
Software Management and Pilot Services

35. T3.4: Exploitation (e-Infrastructure level)
EGI AppDB VMops Dashboard (using IM and OCCI) proving to be big success in production
EGI DataHub (Onedata) developed and maintained as the de facto federating data solution in EGI
ICOS, EOPoC and PanCancer (EOSCpilot demonstrator) actively testing Onedata. Others interested.
Other products are available for exploitation by user communities in the EGI distribution channels and also planned in EOSC-hub thematic platforms
INDIGO-DataCloud Final Review
Software Management and Pilot Services

36. T3.4 Facilitating Contributions upstream
Giacinto’s
From
Presentation
T3.4 Facilitating Contributions upstream
TOSCA adaptor for JSAGA (
OCCI implementation for OpenStack (
Extended AWS support for rOCCI in OpenNebula. Python and Java libraries for OCCI support.
CDMI and QoS extensions for dCache (
Workflow interface extensions for Ophidia (
OpenID Connect Java implementation for dCache (
MitreID ( and OpenID Connect ( libraries
FutureGateway (
OpenStack (
Nova Docker
Heat
OpenID-Connect for Keystone
Pre-emptible instances support (under discussion)
OpenNebula (
OneDock
Patch to support FaSS
Infrastructure Manager (
Clues (
Onedata (
Apache Libcloud (
Kepler Workflow Manager (
The processes and procedures implemented by WP3 facilitated contributions and merge in upstream projects
Giacinto Donvito
INDIGO-DataCloud -- Final Review

37. Software Management and Pilot Services
T3.4: Success stories
Components and features pushed upstream
Heat-translator, OpenStack AAI and CLIs,...
Adoption by industry
TOSCA with Heat translator integrated with IBM Bluemix
Onedata (T-Systems and ATOS)
PaaS services: IM, Orchestrator, etc (T-Systems and ATOS)
Community interest due to openness, SQA and GitHub exposure
udocker (> 200 stars on GitHub) integrated by 4 products, merged community contributions, used in production by Mastercode collab, DisVis and Powerfit
Ophidia used by Lawrence Livermore and Oak Ridge laboratories
Products being pushed to e-infrastructure Distribution Channels (EGI)
INDIGO-DataCloud Final Review
Software Management and Pilot Services

38. Software Management and Pilot Services
Summary
INDIGO-DataCloud Final Review
Software Management and Pilot Services

39. Software Management and Pilot Services
Metrics and KPIs
Metric
Value
Adherence to a code style standard
100%
Adherence to community de-facto standards
Percentage of products following well-known, community de-facto standards
96.86%
Average number of documented code style guidelines
Mean value of code style rules fulfilled by the products
67.66
Average Unit testing coverage
66.69%
Average Automation index
100% means having 1) code style, 2) unit testing, 3) functional testing
75%
Metric
Value
Number of products with Documentation in Markup language
100%
Number of Automated Deployment modules developed and published
49 Ansible Galaxy roles
10 PuppetForge manifests
Number of Continuous Delivery pipelines in Jenkins CI 38
GitHub statistics
196 repositories
84 contributors
12 outside collaborators
~3K pull requests
~1M lines of code(core products)
Upstream contributions
Commits merged, catalogue addition
6 OpenStack products
1 OpenNebula adoption
1 OpenID Connect certified
9 other
DockerHub
indigodatacloud 37 repositories
indigodatacloudapps 22 repositories
INDIGO-DataCloud Final Review
Software Management and Pilot Services

40. Software Management and Pilot Services
Processes and services implemented and operated as planned
Two major
releases as
scheduled
github
Source code
Service Providers Board
External providers
Exploitation
Interface with service providers
Infrastructures and marketplaces
User community feedback
Support product exploitation
Software Quality assurance
Code style checking
Unit testing
Functional/Integration testing
Documentation
Code review
Process outline
Bugtracker
Schedule
Artifacts
SQA
Integration infrastructure
Gerrithub
Gitbook
Jenkins
Metrics/Viz grimoire
SQA results
Software Release and Maintenance
Release Management
Publish release schedules
Manage project public repositories
Continuous Integration
Continuous Delivery
Software maintenance & support
Change Management process
Support to released software.
Problem Management process
Artifacts repos
Support requests
Openproject
Engineering Management Team
Software Release Maintenance
Product teams (WP4,5,6)
GGUS
helpdesk
Exploitation
outcomes
Bugtracker
Maintenance and validation
Software Maintenance and Support
Preview and Development
Infrastructures
Stage Rollout
Adoption by
Communities
Adoption
Upstream
Software Quality Assurance
Testing by
Use cases
Deployment by Infrastructures
Adoption by
Infrastructures
Integration in other products
INDIGO-DataCloud Final Review
Software Management and Pilot Services

41. Better Software for Better Science.
Thank you
Better Software for Better Science.
@indigodatacloud
INDIGO-DataCloud Final Review
Software Management and Pilot Services