Strategy and Policy Officer EGI: advanced computing for research Roberta Piscitelli Strategy and Policy Officer roberta.piscitelli@egi.eu www.egi.eu The EGI-Engage project is co-funded by the European Union (EU) Horizon 2020 program under grant number 654142

EGI: advanced computing for research EGI is a federation of over 300 computing and data centres spread across 56 countries in Europe and worldwide EGI delivers services to support scientists, multinational projects and research infrastructures from public and private sector. EGI is a federation of over 300 computing and data centres spread across 56 countries in Europe and worldwide EGI delivers services to support scientists, multinational projects and research infrastructures www.egi.eu

EGI’s mission is to create and deliver open solutions for science and research by federating digital capabilities, resources and expertise between communities and across national boundaries. EGI’s mission is to create and deliver open solutions for science and research by federating digital capabilities, resources and expertise between communities and across national boundaries.

EGI Foundation 22 national e-Infrastructure providers 1 EIRO: CERN E-Infrastructures are geographically distributed computing resources and data storage facilities linked by high-performance networks. They allow scientists to share information securely, analyse data efficiently and collaborate with colleagues worldwide. They are an essential part of modern scientific research and a driver for economic growth. EGI was established in 2010 building on over a decade of investment by national governments and the European Commission. EGI is a European-wide federation of national computing and data storage resources. Its aim is to support cutting-edge research, innovation and knowledge transfer in Europe. EGI federates resources from various resource centres, mainly from research insitututes and universities. These centres provide computer clusters, storage servers, applications and human support services for secure access and sharing. EGI provides these services to European researchers and their international collaborators. EGI is coordinated by EGI.eu, a not-for-profit foundation based in Amsterdam and owned by EGI’s participants, the National Grid Infrastructures (NGIs). The Council is the supervisory authority and monitors the general course of affairs in the Foundation. It consists of participants and associated participants of the foundation. https://www.egi.eu/about/EGI.eu/council_members.html www.egi.eu/about/egi-foundation/

The largest distributed compute e-Infra worldwide EGI Federation, 2016 QR4 The largest distributed compute e-Infra worldwide 23 Cloud providers >300 HTC data centres 1.7 Million jobs/day 2.6 billion CPU hours/year (+26%) ~50K users (+25%)

A global system of e-Infrastructures Africa-Arabia, Asia and Pacific region, China, Europe, India North and South America, Ukraine China: Institute of HEP, Chinese Academy of Sciences Europe We have several collaboration partnerships with different infrastructures Africa and Arabia: Council for Scientific and Industrial Research, South Africa Latin America: Universida de Federal do Rio de Janeiro India: Centre for Development of Advanced Computing Ukraine: Ukrainian National Grid USA Canada

Serving researchers and innovators i.e. WLCG CTA ELIXIR EPOS EISCAT_3D BBMRI CLARIN LOFAR EMSO ELI LifeWatch ICOS CORBEL ENVRIplus Size of the groups i.e. VRE projects WeNMR DRIHM VERCE MuG AgINFRA CMMST LSGC neuGRID We serve researchers and innovators from public and private sector i.e. Agroknow CloudEO CloudSME Ecohydros Sinergise SixSq Thales (TEISS) Terradue Ubercloud Research Stories … http://go.egi.eu/cst ESFRIs, FET flagships Multinational communities Individual or small groups of scientists Industry and SMEs

A wide offer of services for Research and Innovation

EGI Live Services Compute Storage and Data Training Cloud Compute : Run virtual machines on demand with complete control over resources Cloud Container Compute : Run Docker containers in a lightweight virtualised environment High-Throughput Compute : Execute thousands of tasks to analyse large datasets Storage and Data Online Storage : Store, share and access your files and their metadata on a global scale Archive Storage : Back-ups for long term and future use in a secure environment Data Transfer : Transfer large sets of data from one place to another Training FitSM training : Learn how to manage IT services with a pragmatic and lightweight standard Training infrastructure : Dedicated computing and storage for training and education

EGI Services in the pipeline Operations Service Monitoring: Monitor a wide range of services across your infrastructures Configuration Database: Manage the configuration information of federated e­-infrastructure assets and their functional relations Data Management Enabling Data federation through OneData: The EGI DataHub is a Data as a Service (DaaS) offering from EGI, which aims to collect and give access to reference data sets. The EGI DataHub prototype currently gives users access to a subset (~10TB) of Sentinel-2 data, using GUI, CDMI and POSIX protocols. The goal of this use case is to give users the ability to access and process Earth Observation data remotely without the need to copy the data to storage local to their computing resources. EGI intends to establish a new way of exploiting Earth Observation data together with key scientific datasets, through services that supply the computing and storage resources needed for data access and exploitation and provide the tools to manage the datasets in a distributed environment. Through the EGI Data Hub and the Federated Data Manager, EGI plans to provide researchers and innovators the capability of accessing and integrating huge volumes of different datasets, thus facilitating data use and re-use by coupling data with analytic tools, software and the computing services required to create new knowledge. Last, but not least, EGI aims also at providing a European-wide long-term storage opportunity for Copernicus data ensuring its high-availability for Researchers and for serving SMEs / Industries operating the DIAS platforms and marketplace guaranteeing their long-term availability and usability and realising economy of scale. We will demonstrate this in a hybrid cloud pilot. Data Hub: Selected big datasets of public interest made exploitable – https://datahub.egi.eu Federated Data Manager: Share, discover, and process data federated from different sources Content Distribution: Deliver data in the most efficient way Security Attribute Management: Manage community membership and expose trusted information Check-In: Handle transparent Single Sign-On from multiple heterogeneous identity providers

EGI Services for Participants Operations Coordination Communications Share your successes at a larger scale ITSM Coordination Ensures professional service management for EGI IT services Community Coordination A joint approach to user engagement Operations Coordination and Support Coordinate activities to ensure seamless operations Security Coordination Enhance local security for a safer global infrastructure Project Management and Planning A joint approach to planning and management Strategy and Policy Development Define common strategies and policies in Europe and worldwide Technical Coordination Progress and innovation through collaboration Accounting Track and report the usage of your services Helpdesk Handle EGI service requests and incidents for distributed support teams Operations Tools Integrate resources and operations Collaboration and Community Management Tools Coordinate activities to ensure seamless operations Configuration Database Manage the configuration information of federated e­-infrastructure Service Monitoring Monitor EGI services and provide operational and business insights Validated Software and Repository Manage high-quality software releases Security Attribute Management: Manage community membership and expose trusted information

Services offered to SMEs and Industry Access Reuse Pilots Proof of concepts Datasets Tools/Software Applications Access the EGI e-infrastructure and platforms  Obtain computing capacity to test workflows, models, and applications that will be part of a future advanced ICT product or service. All with dedicated support and consultancy. Reuse open research data sets, tools and applications for product or service development Make the most of the increasing amount of research data sets connected to EGI resources to assist in building your own added-value services. Co-design new products and services Team up with experts on distributed computing systems and all scientific domains to deploy technological solutions using distributed infrastructure or co-develop new products and services. Market your services Benefit from EGI’s events, publications and international network. Get recognition within the EGI ecosystem. EU visibility Networking Market Co-design Products/Services Integrate Share expertise https://www.egi.eu/business/

EGI Catalogue & Use Cases

Cloud Compute With Cloud Compute you can: Run virtual machines on-demand with complete control over the computing resources With Cloud Compute you can: Execute compute- and data-intensive workloads Host long-running services (e.g. web servers or databases) Create disposable testing and development environments Select virtual machine configurations to fit your requirements Manage your Cloud Compute resources in a flexible way with integrated monitoring and accounting capabilities Cloud Compute provides on-demand computing resources to run any kind of workload on virtual machines. You will have complete control over the resources you choose to run a scientific application. See Cloud Compute for service information and request

Powered by Cloud Compute DRIHM Project prototype an e-infrastructure to simulate extreme weather events EXTraS Project implement four software pipelines to harvest data collected on-board ESA’s space observatory XMM-Newton.

Powered by Cloud Compute When a human cell meets Salmonella K. Förstner, Univ. Würzburg, used Cloud Compute to run a pipeline for the analysis of sequencing data. Cloud Compute helped the team to handle demand peaks and that sped up the whole process significantly.  Then they analysed the combined RNA from the both organisms by using so called high-throughput sequencing. Nature (doi:10.1038/nature16547)

Run Docker containers in a lightweight virtualised environment Cloud Container Compute Run Docker containers in a lightweight virtualised environment Main features of Cloud Container Compute: On-demand provisioning Lightweight environment for maximised performance Standard interface to deploy on multiple service providers Interoperable and transparent Removes friction between development and operations environments. A container is a lightweight virtual machine. While a VM is usually made abstracting the full hardware stack (BIOS, CPU, Disk, Network...), the container is built more "directly" on top of the operating system. See Cloud Container Compute for service information and request

Execute thousands of computational tasks to analyse High-Throughput Compute Execute thousands of computational tasks to analyse large datasets Main features of High-Throughput Compute: Access to high-quality computing resources Integrated monitoring and accounting tools to provide information about the availability and resource consumption Workload and data management tools to manage all computational tasks Large amounts of processing capacity over long periods of time The High-Throughput Compute (HTC) service gives access to the EGI infrastructure. With HTC you can, for example, analyse large datasets and execute thousands of parallel computing tasks. HTC computing resources are provided by a distributed network of computing centres, accessible via a standard interface and membership of a virtual organisation See High-Throughput Compute for service information and request

Cherenkov Telescope Array: the world’s leading gamma-ray observatory 1350 scientists from 32 countries CTA EGI usage (2013-2016): 360 million HS06 CPU hours 11 PB of data transferred 2 PB currently in storage 11 million compute jobs Credit: Akihiro Ikeshita Mero-TSK CTA used EGI’s High-Throughput Compute and Online Storage services to manage its computational challenges during the array preparatory phase. http://go.egi.eu/cta

Store, share and access your files and their metadata on Online Storage Store, share and access your files and their metadata on a global scale Main features of Online Storage: Assign global identifiers to files Access highly-scalable storage from anywhere Control the data you share Organise your data using a flexible hierarchical structure Online Storage allows you to store data in a reliable and high-quality environment supported by the EGI resource providers. You can save your data using files and folders in a flexible hierarchical structure. See Online Storage for service information and request

285 Petabytes 285,000,000 GB 11,400,000 Blu-ray Disc Online Storage See Online Storage for service information and request

Back-up your data for the long term and future use in Archive Storage Back-up your data for the long term and future use in a secure environment Main features of Archive Storage: Store large amount of data Free up your online storage Store data for long-term retention Archive Storage allows you to store large amounts of research data in a secure environment, freeing up your usual file storage resources. All the files in Archive Storage are easily located and retrieved to and from different types of platforms. See Archive Storage for service information and request

280 Petabytes 280,000,000 GB 11,200,000 Blu-ray Disc Archive Storage See Archive Storage for service information and request

Transfer large sets of data from one place to another Data Transfer Transfer large sets of data from one place to another Main features of Data Transfer: Ideal for very large files Able to handle large amounts of files Transfer process with automatic retry Data Transfer allows you to move research data from one site to another with dedicated interfaces to display statistics of on-going transfers and manage network resources. File Transfer is ideal to move large amounts of files or files too large to be efficiently handled by common transfer systems. “The most critical infrastructure and tools are the networks and the online storage and data transfer services (…).  Without these the analysis of the data from the LHC would be almost impossible.“ Ian Bird, WLCG project leader See Data Transfer for service information and request

WLCG: Online Storage and Data Transfer at unprecedented scales In 2016, the Worldwide LHC Computing Grid (WLCG) transferred on average 80 Petabytes of data per month. This corresponds to more than 1 billion files per month transferred to thousands of particle physicists working across the world. ATLAS control room at CERN The WLCG is the largest resource provider and service consumer of the EGI Federation

Learn how to manage IT services with a pragmatic and FitSM Training Learn how to manage IT services with a pragmatic and lightweight standard With FitSM Training you can: Increase your expertise in managing IT services Raise your professional profile by a recognised certification 6-9 June 2017 Foundation & Advanced Service Planning and delivery 19-23 June 2017 Advanced Service Operation and Control & Expert FitSM is a lightweight standards family aimed at facilitating service management in IT service provision, including federated scenarios. FitSM training aims at providing those involved in operating federated infrastructures with the professional skills they need in order to effectively manage their services. FitSM professional training is certified by TÜV SÜD, a global leader in standardisation and certification. The qualification programme offers three training levels: Foundation, Advanced and Expert. “I learned how to implement FitSM in an IT organisation and gained from the benefits the framework provides for efficient service management. Pavel Weber, KIT See FitSM Training for service information and request

Dedicated computing and storage for training and education Training Infrastructure Dedicated computing and storage for training and education Main features of the Training Infrastructure Target-specific courses and added value for scientific communities Easy-to-use, on-demand access and improvements in the training offer Allows easy deployment of courses and reuse Training Infrastructure offers a baseline cloud infrastructure for training courses about scientific software and services. Trainers can deploy custom Virtual Machine images on the training infrastructure before the training, and these images offer the training environment for the students. Students can have dedicated training environments, and the community can benefit from the easy deployment, predictability and repeatability of courses, thanks to the cloud-based operational model. See Training Infrastructure for service information and request

EGI services related sessions Tuesday room C1: AAI services Wednesday room C3 Operational Tools Federated Cloud GPGPU Compute Wednesday room F1 EGI DataHub use case: Earth Observation Workshop Thursday room C3 Storage and data services

Discover more Visit www.egi.eu/ Services www.egi.eu/services/ Publications www.egi.eu/about/publications/ Contact us: For public institutions: contact@egi.eu For private organisations: business@egi.eu