SA2 Knowledge Commons for service co-design Gergely Sipos Customer and Technical Outreach Manager, EGI.eu SA2 Activity Manager, EGI-Engage

Outline Overview Achievements Use of resources, problems Plan for PY2 Objectives, tasks, partners and effort Context – SA2 in EGI, in EGI-Engage, in Europe Achievements TSA2.1 Training TSA2.2 Technical User Support (new communities) TSA2.3-10 Competence Centres Use of resources, problems EISCAT_3D portal DARIAH search and information retrieval application Plan for PY2 Summary SA2 Knowledge Commons

Overview

SA2 Partners and effort 47 Participants (incl. Asia Pacific and US) Task Leader / Partner SA2.1 Training Gergely Sipos / EGI.eu SA2.2 Technical User Support Enol Fernandez / CSIC SA2.3 ELIXIR Kimmo Mattila / CSC SA2.4 BBMRI Petr Holub / BBMRI-Eric SA2.5 MoBrain Alexandre Bonvin / BCBR – SURFsara SA2.6 DARIAH Davor Davidovic / RBI – SRCE SA2.7 LifeWatch Jesus Marco de Lucas / CSIC SA2.8 EISCAT_3D Ingemar Häggström / EISCAT SA2.9 EPOS Daniele Bailo / INGV – INFN SA2.10 Disaster Mitigation Simon Lin / AS 47 Participants (incl. Asia Pacific and US) Project Total effort 456.5 PMs 15.2 FTEs Provided by PO SA2 Knowledge Commons

SA2 Partners and effort 47 Participants Project Total effort 456.5 PMs Task Leader / Partner SA2.1 Training Gergely Sipos / EGI.eu SA2.2 Technical User Support Enol Fernandez / CSIC SA2.3 ELIXIR Kimmo Mattila / CSC SA2.4 BBMRI Petr Holub / BBMRI-Eric SA2.5 MoBrain Alexandre Bonvin / BCBR – SURFsara SA2.6 DARIAH Davor Davidovic / RBI – SRCE SA2.7 LifeWatch Jesus Marco de Lucas / CSIC SA2.8 EISCAT_3D Ingemar Häggström / EISCAT SA2.9 EPOS Daniele Bailo / INGV – INFN SA2.10 Disaster Mitigation Simon Lin / AS 47 Participants Project Total effort 456.5 PMs 15.2 FTEs Provided by PO SA2 Knowledge Commons

SA2 (WP6) objectives Objective 1. Identify and support communities and users from EGI and its partners 2. Facilitate the integration of scientific applications with EGI’s e-Infrastructure services 3. Co-design and co-develop services for sustainable, structured scientific communities 4. Promote and support the uptake of new services within scientific communities 5. Provide a training framework, foundational training services and domain specific training events for scientific communities, EGI members and partners list of relevant project objectives for the WP SA2 Knowledge Commons

SA2 in EGI: The EGI Engagement pipeline http://go. egi SA2 in EGI: The EGI Engagement pipeline http://go.egi.eu/engagementstrategy (D2.2, 2.8) Outreach and training Technical support Requirement collection Co-development Testing and evaluation Dissemination, exploitation Arrangements for long-term operation Long-term operation  SA2.1 Training  SA2.2 Tech. User Support SA2.3-10 – Comp. Centres 1 Training (SA2.1) 2 3 New communities (SA2.2) 4 5 6 7 8 ELIXIR (SA2.3) BBMRI (SA2.4) MoBrain (SA2.5) . . . SA2 Knowledge Commons

SA2 in EGI-Engage: Partnerships within the project Dissemination, Exploitation NA2.1 Communication and Dissemination NA2.2 Strategy, Business Development and Exploitation JRA1.3 Accounting JRA1.4 Operations Tools JRA1.5 Resource Allocation JRA2.1 Federated Open Data (in PY2) JRA2.2 Federated Cloud JRA2.4 Accelerated Computing SA1.3 Integration, Deployment of Grid and Cloud Platforms SA2 Requirements Services SA2 Knowledge Commons

SA2 in Europe: Partnerships beyond EGI-Engage SA2.1 SA2.2 SA2.3 SA2.4 SA2.5 SA2.6 SA2.7 SA2.8 SA2.9 SA2.10 EUDAT2020, ENVRIplus, AARC, Indigo-Datacloud, … ELIXIR-EXCELERATE BBMRI Common Services IT INSTRUCT, West-Life VRE DARIAH e-infrastructure WG LTER, GBIF Supporting EISCAT_3D (NeIC), DIRAC EPOS-RI LTER: The Long Term Ecological Research Network GBIF: Global Biodiversity Information Facility SA2 Knowledge Commons

Achievements Main achievements of the work package

Training Task 6.1 Cloud-based training infrastructure For cloud trainings AND for application trainings Already used by 9 events (5 in Bari), 945 VMs instantiated New element in the EGI Service Portfolio Reference course for Federated Cloud Delivery in 5 countries for approx. 100 people Security training courses For site admins: Securing e-infrastructure sites For users: ‘Dos and don’ts in VM preparation’ Event support 18 events and 3 webinars Joint activities with EUDAT Defined ‘reference use case’ – currently under implementation (JRA2) Joint events: EUDAT training session in Bari (November 2015) ‘Design your e-infrastructure’ workshop in Amsterdam (April 2016) SA2 Knowledge Commons

Technical user support Task 6.2 Expanded support network and support tools Continued ‘EGI engagement’ meetings Ticketing queue with supporters’ manual to track support cases Introduction of ‘Federated cloud user support network’ 18 teams,12 countries, 14 meetings Target groups RIs, Future and Emerging Technologies (FET) - 19 (besides the CCs!) H2020 projects and structured communities – 11 Long-tail of science  brokered to national teams SME and industry  by NA2 Popular topics (Federated) community cloud Containers, orchestrators Data sharing, data access ‘Community-specific AAI’ Spread across disciplines, incl. CCs (D2.8 includes details): Biological & medical (incl. biodiversity, ecosystem): 47.4% (18/38) Earth sciences: 21% (8/38) Physics (incl. astro, particle, laser, etc.): 21.2% (8/38) Digital humanities (incl. languages): 5.2% (2/38) Agriculture: 2.6% (1/38) Nanotechnology: 2.6% (1/38) SA2 Knowledge Commons

Support for new communities – Some examples ICOS – Coupling compute with EUDAT storage Euro-Argo – ‘Data subscription personal cloud’ ASTRON IBM research lab – network topology research on MPI clusters SKA and LOFAR –adoption of cloud services … BILS (Bioinformatics Infrastructure for Life Science) PconsC2, Pcons, TOPCONS2 servers on Federated Cloud TOPCONS2 predicted more than 2 million protein sequences by 5000 users from 69 countries 1 2 3 4 5 6 7 8 Outreach Tech.supp Reqs. Co-devel. Testing Exploit. SLA arran. Operation Human Brain Project Access to brain images with Federated Open Data (JRA2) 10 TB dataset hosted at CYFRONET, DESY Peachnote–Search inside music scores Search engine hosted on Jüelich cloud site Tens of thousands of users every day - through IMSLP, the largest sheet music library SA2 Knowledge Commons

Support for new communities – Some examples NEW CC in PY2 EMSO Setup Hadoop clusters on EGI OpenStack sites with Sahara Implemented on INFN-Bari cloud site ICOS – Coupling compute with EUDAT storage Euro-Argo – ‘Data subscription personal cloud’ ASTRON IBM research lab – network topology research on MPI clusters SKA and LOFAR –adoption of cloud services … BILS (Bioinformatics Infrastructure for Life Science) PconsC2, Pcons, TOPCONS2 servers on Federated Cloud TOPCONS2 predicted more than 2 million protein sequences by 5000 users from 69 countries 1 2 3 4 5 6 7 8 Outreach Tech.supp Reqs. Co-devel. Testing Exploit. SLA arran. Operation Human Brain Project Access to brain images with Federated Open Data (JRA2) 10 TB dataset hosted at CYFRONET, DESY Peachnote–Search inside music scores Search engine hosted on Jüelich cloud site Tens of thousands of users every day - through IMSLP, the largest sheet music library SA2 Knowledge Commons

Distributed Competence Centre Competence Centres: SA2.3-10 ESFRI Landmark Distributed Competence Centre BBMRI DARIAH EISCAT-3D ELIXIR EPOS LifeWatch MoBrain Disaster mitigation EMSO (with MoU) NEW! ESFRI Landmark ESFRI Landmark ESFRI Project ESFRI Projects: The ESFRI Projects have been selected for scientific excellence and maturity and are included in the Roadmap in order to underline their strategic importance for the European Research Infrastructure system and support their timely implementation. The ESFRI Projects can be at different stages of their preparation according to the date of inclusion in the ESFRI Roadmap. ESFRI Landmarks: The ESFRI Landmarks are the RIs that were implemented or started implementation under the ESFRI Roadmap and are now established as major elements of competitiveness of the European Research Area. The ESFRI Landmarks need continuous support for successful completion, operation and upgrade in line with the optimal management and maximum return on investment. ESFRI Landmark ESFRI Landmark ESFRI Landmark ESFRI Project SA2 Knowledge Commons

Competence Centres: Diverse interests, different timelines Number of relevant EGI service areas Several, interlinked areas EPOS ELIXIR MoBrain AAI Federated cloud Federated Open Data Application porting Core platform (AAI, Service registry) Federated Cloud VM Marketplace Application/tool porting GPGPUs Federated cloud Community platform Application porting DARIAH Disaster Mitigation LifeWatch Few, independent areas DataHub Real-time search application Compute apps Data-intensive computing Federated cloud Community portals R and citizen tools Federated cloud Community VREs RIs on the right are more strongly connected to NGIs, RIs in the left are much less BBMRI EISCAT_3D Private clouds Controlled data sharing One/two areas Data catalogue HTC Service maturity target (by end of EGI-Engage) Pilot exploration of e-infra services Production e-infra services SA2 Knowledge Commons

CCs achievements in PY1 Implementation plans; Software tools/services 9 milestones and deliverables in PY1 Requirements and implementation plans for Software package to deploy private cloud at a biobank (BBMRI) Life science applications – 5 (ELIXIR) Earth science use cases – 3 (EPOS) Community application: Image-based flora recognition (Pattern recognition) (LifeWatch) R tools for ecosystem modelling (LifeWatch) Data repository for digital humanities (DARIAH) Real-time search and information retrieval (DARIAH) User portals EISCAT MoBrain SA2 Knowledge Commons

CCs in the Engagement pipeline Requirement collection & system design Co-development Tests, evaluation Dissemin./ Exploitation SLAs, operation ELIXIR Compute Platform with AAI 5 use cases (M6.3) Ongoing BBMRI Private clouds (M6.2) Data workflows MoBrain Community portal Done Done (D6.4) Started (ISGC, Utrecht, etc.) SLA-OLA signed in Q1 2016 Cryo-EM in the cloud (SCIPION) GPGPU application gateways DARIAH Data repository Done (D6.2) Real-time search Done/Ongoing (D6.5) Application gateway LifeWatch GPU & cloud infra. deployment Apps. for citizen science Done (D6.1) EISCAT_3D Data Portal Done–Proof-of-concept (D6.3) Data & Compute Portal EPOS 3 use cases (M6.4) Disaster Mitigat. ELIXIR 2 simulation gateways gWRF and iCOMCOT gateways 3 core use cases Typhoons (Haiyan, Morakot) 2014 rainfall in Malaysia Thailand flood in 2011 2 external use cases (NE, VT) WP6 Knowledge Commons

What have the PY1 outcomes enabled? ELIXIR: Implementation of ‘ELIXIR Compute Platform’; Deployment exists at 2 sites, ongoing at 3 more sites BBMRI: Software package for private biobank cloud installations (IaaS+PaaS); Deployments & workflows in PY2 MoBrain Application portals to perform multi-scale simulations on mixed CPU and GPU systems DARIAH Demonstrators to capture the ‘cloud interest’ of digital humanities; Example use cases from the domain LifeWatch Reaching citizen scientists, with demonstrated use at Real Jardín Botánico; Consolidation of R tools EISCAT_3D Initial data schemas and data access & management services for EISCAT_3D EPOS An ecosystem with access control, data sharing, cloud, thematic exploitation platforms Disaster Mitigation Re-simulating typhoon events that caused 7,000 fatalities and 9 billion USD damage (in 2009, in 2013) ELIXIR: - Deployment finished at CESNET and GRNET; Ongoing at CSC, EBI, FRANCE SA2 Knowledge Commons

Building the ‘Knowledge Commons’ People’s knowledge Brought together approx. 70 experts from 50 institutes EMSO joining as unfunded Competence Centre in PY2 21 training events (f2f and webinar) International network of cloud user support teams Technical knowledge 25 software service for/by competence centres Each with user and technical documentations; Open licensed source code Infrastructure knowledge 8 discipline-specific infrastructure Virtual Organisations 2 generic Virtual Organisations (training and fedcloud.egi.eu) SA2 Knowledge Commons

Use of Resources and Problems Numbers will be provided by PO, work package leaders must provide explanation

REPORTED in PY1 (% of total) EXPLANATION OF DIFFERENCE Use of resources TASK TOTAL EFFORT M1-30 (PMs) REPORTED in PY1 (% of total) EXPLANATION OF DIFFERENCE SA2.1 – Training 37.0 23% Broadening activity in PY2-3 SA2.2 – Tech. User Support 56.5 41% Uniform intensity SA2.3 - ELIXIR M7-M24 36.5 13% Started in M7 with requir. collection SA2.4 - BBMRI M7-M30 29.0 34% Started in M7 SA2.5 - MoBrain 59.5 59% Parallel development activities in PY1 SA2.6 – DARIAH 62.5 51% SA2.7 - LifeWatch 60.0 29% Broadening activities in PY2-3 SA2.8 - EISCAT_3D 28.0 26% Technology pathfinding activity in PY1 SA2.9 - EPOS 30.5 18% Start with requirement collection. SA2.10 – Disaster Mitigation 57.0 (unfunded) Use case pathfinding activities in PY1 SA2 Knowledge Commons

Problems EISCAT_3D portal (D6.3) Baseline portal technology had to be replaced Mitigation: External technology was brought in: DIRAC Proof-of-concept portal setup released in March Positive user evaluation  Effort reallocation for PY2 Adding computing capabilities in 2nd portal version: next 18 months DARIAH Search and Information Retrieval application (D6.5) Development took more time and effort than planned Full version will be available in Q2 2016. (Instead of end of Feb) Unfunded extra effort by GWDG (Germany) SA2 Knowledge Commons

Plan for PY2

Plans PY2 SA2.1: Training SA2.2: Technical user support Emerging solutions: GPGPU platform, Federated Data Manager, DataHub, IdP proxy, long tail platform CC services and demonstrators EGI-EUDAT reference course Further expansion of cloud training portfolio (PaaSs, Docker, APIs) SA2.2: Technical user support Follow-up active use cases – for example ‘Design your e-infrastructure workshop’ Active use of new platform for the long tail of science Engage with additional communities New opportunities in ESFRI roadmap 2016 New opportunities by H2020 projects – e.g. VREs, RI-related projects SA2 Knowledge Commons

PY2 plans – Competence Centres Requirement collection & system design Co-development Tests, evaluation Dissemin./ Exploitation SLAs, operation ELIXIR Compute Platform with AAI 5 use cases (M6.3) Ongoing BBMRI Private clouds (M6.2) Data workflows MoBrain Community portal Done Done (D6.4) Started (ISGC, Utrecht, etc.) SLA-OLA signed in Q1 2016 Cryo-EM in the cloud (SCIPION) GPGPU application gateways DARIAH Data repository Done (D6.2) Real-time search Done/Ongoing (D6.5) Application gateway LifeWatch GPGPU, Cloud infra. Services Apps. for citizen science Done (D6.1) EISCAT_3D Data Portal Done–Proof-of-concept (D6.3) Data & Compute Portal EPOS 3 use cases (M6.4) Disaster Mitigat. ELIXIR 2 simulation gateways gWRF and iCOMCOT gateways 3 core use cases Typhoons (Haiyan, Morakot) 2014 rainfall in Malaysia Thailand flood in 2011 2 external use cases (NE, VT)

PY2 plans – Competence Centres Requirement collection & system design Co-development Tests, evaluation Dissemin./ Exploitation SLAs, operation ELIXIR Compute Platform with AAI 5 use cases (M6.3) Operates in 5 ELIXIR nodes Completed (D6.10, D6.15) EXCELERATE Jointly by ELIXIR and EGI BBMRI Private clouds (M6.2) Operated in 3 biobanks Completed (D6.11) Common IT serv. Data workflows (D6.8) Ongoing MoBrain Community portal Done Done (D6.4) SLA-OLA signed in Q1 2016 Cryo-EM in the cloud (SCIPION) Done (D6.14) GPGPU application gateways Completed (D6.7, D6.12) Completed DARIAH Data repository Done (D6.2) Real-time search Done/Ongoing (D6.5) Application gateway Completed (D6.17 LifeWatch GPGPU, Cloud infra. Services GPGPU apps in production Apps. for citizen science Done (D6.1, D6.18) EISCAT_3D Data Portal Done–Proof-of-concept (D6.3) Data & Compute Portal EPOS 3 use cases (M6.4) Completed (D6.13, D6.19) Ongoing(Compute) Ongoing (Data) Disaster Mitigat. ELIXIR 2 simulation gateways gWRF and iCOMCOT gateways Completed (D6.9) 3 core use cases Typhoons (Haiyan, Morakot) Completed (D6.20) 2014 rainfall in Malaysia Thailand flood in 2011 2 external use cases (NE, VT) Completed (Nepal, Vietnam) SA2.3-10: 13 deliverables in next 12 months

Summary Objective 1 (O1): The continued coordination of the EGI Community Continued coordination of ‘EGI Engagement’ activity Deeper collaborations: 27 RIs/FETs, 11 communities, numerous long-tail users Nurtured the network of Federated Cloud user support teams Objective 5 (O5): Promotion the adoption and extension of the current EGI services Designed EGI-based community platforms for ELIXIR, BBMRI, EPOS Launched co-developments of EGI-based services for 8 ESFRI communities Supported/delivered 21 training events Reached new users: from citizen science ecologists to HBP, EMSO, SKA SA2 Knowledge Commons

Next: Jesús Marco de Lucas, IFCA-CSIC (Spain) Coord. of LifeWatch Competence Center