1. JRA1 E-Infrastructure Commons
Diego Scardaci
Technical Outreach Expert, EGI.eu/INFN
WP3 Activity Manager, EGI-Engage

2. JRA1 E-Infrastructure Commons
Outline
Overview
Objectives, tasks, partners and effort
Achievements
JRA1 activities & the EGI Platform Architecture
AAI
Marketplace
Accounting
Operations Tools
Resource allocation – e-GRANT
Use of resources
Plan for PY2
Summary
JRA1 E-Infrastructure Commons

3. Overview

4. JRA1 E-Infrastructure Commons
JRA1 (WP3) objectives
Tasks
Task Objectives
TJRA1.1
Provide viable methods for authentication and authorisation in the EGI ecosystem.
TJRA1.2 TJRA1.5
Simplify the access to the infrastructure services through innovation and new services in the area of Service Registry, Marketplace and resource allocation.
TJRA1.3
Evolve the EGI accounting system to manage the data deluge expected over the next years, including new types of accounting metrics (e.g. data accounting) and redesign of the presentation layer to improve the user experience
TJRA1.3 TJRA1.4
Adapt the operations tools to new technologies to satisfy requirements from service providers and user communities.
TJRA1.1 TJRA1.2 TJRA1.3 TJRA1.4 TJRA1.5
Define interfaces to create a network of analogue tools that provides users with integrated view of the infrastructures
list of relevant project objectives for the WP
JRA1 E-Infrastructure Commons

5. JRA1 partners and effort
13 Participants
PY1 effort
61 PMs
Project Total effort
153 PMs
5.1 FTEs
JRA1
Task
Leader / Partner
JRA1.1
Authentication and Authorisation Infrastructure
Christos Kanellopoulos / GRNET
JRA1.2
Service Registry and Marketplace
Dean Flanders / SWING
JRA1.3
Accounting
Stuart Pullinger / STFC
JRA1.4
Operations Tools
Cyril L'Orphelin / CNRS
JRA1.5
Resource Allocation – e-GRANT
Tomasz Szepieniec / CYFRONET
Provided by PO
JRA1 E-Infrastructure Commons

6. Achievements
Main achievements of the work package

7. JRA1 Activities & The EGI Platform Architecture
Community Platforms
Brokering, community-specific data, tools and applications
Marketplace
e-GRANT
EXPAND
EGI endorsed VM images, Helpdesk
Collaboration Platform
VM Image Catalogue of
Data-intensive computing
Cloud
GPGPU Platform
Open Data Platform
OPERATE
& DEVEL
OPERATE
NEW
NEW
EGI Core Infrastructure Platform
AAI, Service Registry, Accounting, Monitoring
Federated Service Management
New AAI
Ops Portal
Accounting
GOCDB
ARGO
Messaging
OPERATE & DEVEL
OPERATE
DEVEL EXPAND
Physical Infrastructure
JRA1 E-Infrastructure Commons

8. Messaging Infrastructure
EGI Core Platform
The EGI Core Infrastructure Platform provides all the tools to operate and manage a distributed infrastructure
EGI Core Platform
Messaging Infrastructure
Service Registry
AAI
Monitoring
Accounting
Information Service
Operations Portal
Single Sign-On and VO management
Catalogue of the services
Status of the infrastructure
Ticketing System
AAI
Service Registry
Messaging Infrastructure
Monitoring
Accounting
Operations Portal
Helpdesk
Services A/R
Usage or resources
VO management & Dashboards
Coord. Activities
Operations Support
Security Coordination
JRA1 E-Infrastructure Commons

9. The new EGI AAI infrastructure TJRA1.1
JRA1 E-Infrastructure Commons

10. AAI Core requirements for the AAI
Credentials/Tokens
Users access EGI services with credentials released by his/her home organisation (eduGAIN support)
Take into account the so-called homeless users
Level Of Assurance (LoA) for each credential type
Open architecture
Support the most common technologies to manage federated identities: SAML, OpenID Connect, X.509, etc.
Support several attributes sources
Easily extensible and interoperable with other infrastructures
Hide the complexity to the Service providers
Token Translator Services (TTSs)
Convert a credential to be recognised by the service
JRA1 E-Infrastructure Commons

11. AAI New EGI AAI and trust model
Level of Assurance
TRUST
Information sent to service providers
Community attributes
“User A”
User
TRUST
EGI Services
Community Attribute
Authority
JRA1 E-Infrastructure Commons

12. AAI Liaison with AARC & Requirements gathering
Collaboration established with the AARC project:
Adopt AAI policies, solutions and best practices defined at European level
Deal with problems that require a larger scope to be resolved (e.g. global unique identifiers, levels of assurance, etc.)
The AARC Blueprint
IdP/SP proxy model, TTS based on CILogon
Requirements gathering
EGI-Engage Competence Centers
Other EGI communities and RIs
EGI Tools
JRA1 E-Infrastructure Commons

13. AAI The new EGI AAI architecture
EGI infrastructure
Attributes
Community Portals
EGI Resources
Token Translators
Attributes
Attributes
User
EGI Tools
Attributes
IdP
IdP/SP Proxy
Attribute Authorities
Attributes SP
Compliant with the AARC
Architecture Blueprint
Social IDs
eGOV ID
JRA1 E-Infrastructure Commons

14. AAI The EGI IdP/SP proxy
EGI Services
Features
Support for LoA
Supported IdPs:
SAML2.0: eduGAIN
OIDC/OAuth2: Google, Facebook, LinkedIn, ORCID
SAML STORK: eGOV Ids
OpenConnext Core Engine (simpleSamlPHP)
IdP SP
User Enrollment
IdP Discovery
User Consent
User
IdP/SP
Proxy
Social IDs
eGOV IDs
JRA1 E-Infrastructure Commons

15. AAI Credential/Token Translator System
CILogon (extended)
PUSP generator
Online CA
Attributes
Community
Portal
Master Portal
User
EGI IdP/SP
Attributes
Attributes
EGI Service
EGI Service
guidelines
O(1) Online CA: infra agnostic
O(10) Master Portals: one per infra, VO agnostic
O(many) Community portals
JRA1 E-Infrastructure Commons

16. AAI Pilot activities with ELIXIR and EPOS RIs
Integration of the ELIXIR IdP with the new EGI AAI
EGI tools: GOCDB, AppDB, Ops Portal, Accounting, Monitoring
EGI computing and storage resources
Consultancy to design the AAI of the EPOS RI
Requirements
User access to the Integrated Core Services (ICS) with different type of credentials
User delegate ICS to fetch resources at the various Thematic Core Services (TCSs)
Collaboration with SA2
EPOS RI architecture
JRA1 E-Infrastructure Commons

17. The EGI Marketplace TJRA1.2
JRA1 E-Infrastructure Commons

18. JRA1 E-Infrastructure Commons
Marketplace Concepts
Ensure the discoverability of the services delivered by EGI
Facilitate service discovery and reuse
Facilitate access to existing expertise
Increase the visibility of providers
Increase competitiveness by providing a low cost of entry
Facilitate inter-disciplinary research
Collaborative improvements of products and services
Compose services in workflows
Concepts defined in collaboration with NA2
JRA1 E-Infrastructure Commons

19. Marketplace Requirements
Gathered via survey and interviews
Sixteen resource providers, projects and research communities
Communities
BSC
VENUS-C
SurfSARA
STFC
ELIXIR
CSC
DARIAH
LSGC
Neugrid
EGI.eu
Cyfronet
iMarine
SZTAKI
WeNMR
France Gr.
GRNET
JRA1 E-Infrastructure Commons

20. Marketplace Assessment of the technologies
3 types of solutions and 6 technologies:
EGI tools to be extended: AppDB, GOCDB
Supported by other initiatives: OpenIRIS and FIWARE
Generic web tools marketplace oriented: WooCommerce, WordPress
Main evaluation metrics:
Requirements
Expertise within the EGI collaboration
Operational and maintenance models
Terms of use
Compliance to policies and regulations
Costs of ownership
JRA1 E-Infrastructure Commons

21. The EGI Accounting System TJRA1.3
JRA1 E-Infrastructure Commons

22. Accounting The EGI Accounting system
Accounting Repository
Service
Probe
Probe
Accounting Portal
JRA1 E-Infrastructure Commons

23. Accounting Evolution of the Accounting Repository architecture
Message Broker
3 Broad categories:
Optimised use of the MySQL database
Replacement with Apache Hadoop and the Hadoop Distributed File System
Replacement with a time series databases
Ingest
Export
Data Store
Export
Ad-hoc queries
Aggregator
The central processing stage for the CPU accounting data takes many hours to complete and operates in a single processor thread. Although the data is backed up regularly, the data is not distributed across multiple hosts to provide greater resilience and processing power. Recent advances in Big Data tools provide an opportunity to address these limitations: improving the performance and resilience of the central repository.
Configuration
Start of testing period
End of testing period
Parallel batch processing
March 2016
April 2016
Replace APEL tools with Hadoop/HDFS tools
May 2016
June 2016
Replacement APEL Backend
July 2016
August 2016
Combined MySQL and Hadoop
September 2016
October 2016
Parallel stream processing
November 2016
December 2016
JRA1 E-Infrastructure Commons

24. Accounting Data Accounting
Extend accounting to account for usage of data sets
Data set as a logical set of files uniquely identified (PID)
Data Accounting
More efficient use of the infrastructure
Assist scientists in assessing the impact of their work
Requirements gathered via survey and interviews
User-access, data set PID logging, and recording transfer operations
First prototype in PY2
Identified solutions/links:
WLCG Computing Resources Scrutiny
EGI Open Data platform (PID system)
Communities
ELIXIR
D4Science
EPOS
WestLife
LifeWatch
WeNMR
ESA TEPs
CESNET
EISCAT
DASSH
BILS/NBIS
IFCA
JRA1 E-Infrastructure Commons

25. Accounting The new Accounting Portal
New look & feel
Dynamic capabilities
Hypertextual navigation
Technologies:
Python, Django
Bootstrap, Dojo
Accounting for scientific disciplines
JRA1 E-Infrastructure Commons

26. Accounting Other Outcomes
Cloud accounting
New fields: number of public IPs, benchmark type and benchmark
Accelerate computing accounting
Analysis of the GPGPU batch system logs
New versions of three batch systems support
HTCondor
Univa Grid Engine
Torque
JRA1 E-Infrastructure Commons

27. JRA1 E-Infrastructure Commons
Operational Tools Ops Portal, Service Catalogue, Monitoring, Messaging TJRA1.4
JRA1 E-Infrastructure Commons

28. Operational Tools The EGI Operations Portal
Portal for operations: broadcast tool, VO management facilities, operation and security dashboards, metrics, etc.
VO Administration and operations PORtal (VAPOR)
Cloud support
Resource browser (based on OGF GLUE2)
VO ID card
VO acknowledgment statement
Portal/Web Service robot certificates
New metric module
Users distribution over disciplines, CAs, Vos
Improvements on ROD and Security dashboards
JRA1 E-Infrastructure Commons

29. Operational Tools The EGI Service Registry - GOCDB
Catalogue to record information about resource providers, services, service-endpoint of a digital infrastructure
Multi-Tenants
Support multiple projects in a unique instance (EUDAT)
Ease extension of the data model
Integration with the new EGI AAI
SAML support
Fine-grained access model
JRA1 E-Infrastructure Commons

30. Operational Tools The EGI Monitoring System - ARGO
Framework for monitoring status, availability and reliability
Availability & Reliability monitoring:
Rack a vast number of monitoring metrics
Provide real-time notifications and status reports
Monitor SLAs/OLAs
Relies on Nagios for status monitoring
Multiple reports using customer defined profiles
Multi-Tenant support
Custom factors in the report generation
New Web API
Monitoring as a Service
JRA1 E-Infrastructure Commons

31. Operational Tools The EGI Messaging Service
Enables reliable asynchronous messaging for the EGI infrastructure (ActiveMQ and STOMP)
Towards a scalable ΗTTP Messaging as a Service:
HTTP API for client access
Transparent scalability & high availability for clients
Multi-tenant support and easy updates
JRA1 E-Infrastructure Commons

32. Resource Allocation – e-GRANT TJRA1.5
JRA1 E-Infrastructure Commons

33. Resource Allocation e-GRANT
Support the EGI Resource Allocation Process by allowing researchers to request compute and storage resources
New version of e-GRANT integrating the Pay-for-Use process
Main features:
List of pools
Find pools
Price negotiation
SLA generation
Pay-for-Use introduced in NA2 presentation
JRA1 E-Infrastructure Commons

34. Use of Resources
Numbers will be provided by PO, work package leaders must provide explanation

35. JRA1 – Effort consumed in PY1
JRA1 E-Infrastructure Commons

36. Plan for PY2
Freedom on way to present

37. JRA1 E-Infrastructure Commons
Plan for PY2
TJRA1.1 - AAI
EGI IdP/SP as eduGAIN service provider
Enrollment service
Interconnect the EGI IdP/SP proxy with an OIDC service (Open Data Platform)
Pilot with use cases selected by the competence centers
TJRA1.2 – Marketplace
Complete the assessment of the short-listed solutions
Demonstrator
Term of reference to publish services (with NA2)
JRA1 E-Infrastructure Commons

38. JRA1 E-Infrastructure Commons
Plan for PY2
TJRA1.3 – Accounting
New accounting repository architecture
Data accounting prototype
Evolution on Cloud and GPGPU accounting
New portal deployed in production
TJRA1.4 – Operations tools
Support of the new EGI AAI
Ops Portal: Integration with other user management services
GOCDB: writable REST API
ARGO: probes management, streaming engine for real-time computations
Messaging: HTTP messaging service
Security Monitoring: automatic framework for VM image assessment
TJRA1.5 – e-GRANT: new version of EGI RA process
JRA1 E-Infrastructure Commons

39. JRA1 E-Infrastructure Commons
Summary (I)
Objective 1 (O1): The continued coordination of the EGI Community.
Coordination of the EGI Core Infrastructure Platform innovation activities
Objective 2 (O2): EGI Solutions, related business models and access policies.
Tailoring the EGI core platform services to the needs of specific groups (CCs)
Objective 3 (O3): Offer and expand an e-Infrastructure Commons solution
New AAI architecture
Enhancements of the accounting and monitoring systems
GOCDB: support multiple projects and easy extension of the data model
EGI Service Registry and Marketplace: user requirements gathered and assessment of the technologies
JRA1 E-Infrastructure Commons

40. JRA1 E-Infrastructure Commons
Summary (II)
Objective 4 (O4): Open data platform and the European Big Data Value.
Data usage accounting
Objective 5 (O5): Promotion the adoption and extension of the current EGI services
User communities involved in the definition of the EGI tool roadmaps (including new services and features)
Achievements grouped by objectives
JRA1 E-Infrastructure Commons