1. SA2 Knowledge Commons for service co-design
Gergely Sipos
Customer and Technical Outreach Manager, EGI.eu
SA2 Activity Manager, EGI-Engage

2. Outline Overview Achievements Use of resources, problems Plan for PY2
Objectives, tasks, partners and effort
Context – SA2 in EGI, in EGI-Engage, in Europe
Achievements
TSA2.1 Training
TSA2.2 Technical User Support (new communities)
TSA Competence Centres
Use of resources, problems
EISCAT_3D portal
DARIAH search and information retrieval application
Plan for PY2
Summary
SA2 Knowledge Commons

3. Overview

4. SA2 Partners and effort 47 Participants (incl. Asia Pacific and US)
Task
Leader / Partner
SA2.1
Training
Gergely Sipos / EGI.eu
SA2.2
Technical User Support
Enol Fernandez / CSIC
SA2.3
ELIXIR
Kimmo Mattila / CSC
SA2.4
BBMRI
Petr Holub / BBMRI-Eric
SA2.5
MoBrain
Alexandre Bonvin / BCBR – SURFsara
SA2.6
DARIAH
Davor Davidovic / RBI – SRCE
SA2.7
LifeWatch
Jesus Marco de Lucas / CSIC
SA2.8
EISCAT_3D
Ingemar Häggström / EISCAT
SA2.9
EPOS
Daniele Bailo / INGV – INFN
SA2.10
Disaster Mitigation
Simon Lin / AS
47 Participants (incl. Asia Pacific and US)
Project Total effort
456.5 PMs
15.2 FTEs
Provided by PO
SA2 Knowledge Commons

5. SA2 Partners and effort 47 Participants Project Total effort 456.5 PMs
Task
Leader / Partner
SA2.1
Training
Gergely Sipos / EGI.eu
SA2.2
Technical User Support
Enol Fernandez / CSIC
SA2.3
ELIXIR
Kimmo Mattila / CSC
SA2.4
BBMRI
Petr Holub / BBMRI-Eric
SA2.5
MoBrain
Alexandre Bonvin / BCBR – SURFsara
SA2.6
DARIAH
Davor Davidovic / RBI – SRCE
SA2.7
LifeWatch
Jesus Marco de Lucas / CSIC
SA2.8
EISCAT_3D
Ingemar Häggström / EISCAT
SA2.9
EPOS
Daniele Bailo / INGV – INFN
SA2.10
Disaster Mitigation
Simon Lin / AS
47 Participants
Project Total effort
456.5 PMs
15.2 FTEs
Provided by PO
SA2 Knowledge Commons

6. SA2 (WP6) objectives Objective
1. Identify and support communities and users from EGI and its partners
2. Facilitate the integration of scientific applications with EGI’s e-Infrastructure services
3. Co-design and co-develop services for sustainable, structured scientific communities
4. Promote and support the uptake of new services within scientific communities
5. Provide a training framework, foundational training services and domain specific training events for scientific communities, EGI members and partners
list of relevant project objectives for the WP
SA2 Knowledge Commons

7. SA2 in EGI: The EGI Engagement pipeline http://go. egi
SA2 in EGI: The EGI Engagement pipeline (D2.2, 2.8)
Outreach and training
Technical support
Requirement collection
Co-development
Testing and evaluation
Dissemination, exploitation
Arrangements for long-term operation
Long-term operation
 SA2.1 Training
 SA2.2 Tech. User Support
SA – Comp. Centres 1
Training (SA2.1) 2 3
New communities (SA2.2) 4 5 6 7 8
ELIXIR (SA2.3)
BBMRI (SA2.4)
MoBrain (SA2.5)
. . .
SA2 Knowledge Commons

8. SA2 in EGI-Engage: Partnerships within the project
Dissemination, Exploitation
NA2.1 Communication and Dissemination
NA2.2 Strategy, Business Development and Exploitation
JRA1.3 Accounting
JRA1.4 Operations Tools
JRA1.5 Resource Allocation
JRA2.1 Federated Open Data (in PY2)
JRA2.2 Federated Cloud
JRA2.4 Accelerated Computing
SA1.3 Integration, Deployment of Grid and Cloud Platforms
SA2
Requirements
Services
SA2 Knowledge Commons

9. SA2 in Europe: Partnerships beyond EGI-Engage
SA2.1 SA2.2 SA2.3 SA2.4 SA2.5 SA2.6 SA2.7 SA2.8 SA2.9 SA2.10
EUDAT2020, ENVRIplus, AARC, Indigo-Datacloud, …
ELIXIR-EXCELERATE
BBMRI Common Services IT
INSTRUCT, West-Life VRE
DARIAH e-infrastructure WG
LTER, GBIF
Supporting EISCAT_3D (NeIC), DIRAC
EPOS-RI
LTER: The Long Term Ecological Research Network
GBIF: Global Biodiversity Information Facility
SA2 Knowledge Commons

10. Achievements
Main achievements of the work package

11. Training Task 6.1 Cloud-based training infrastructure
For cloud trainings AND for application trainings
Already used by 9 events (5 in Bari), 945 VMs instantiated
New element in the EGI Service Portfolio
Reference course for Federated Cloud
Delivery in 5 countries for approx. 100 people
Security training courses
For site admins: Securing e-infrastructure sites
For users: ‘Dos and don’ts in VM preparation’
Event support
18 events and 3 webinars
Joint activities with EUDAT
Defined ‘reference use case’ – currently under implementation (JRA2)
Joint events:
EUDAT training session in Bari (November 2015)
‘Design your e-infrastructure’ workshop in Amsterdam (April 2016)
SA2 Knowledge Commons

12. Technical user support Task 6.2
Expanded support network and support tools
Continued ‘EGI engagement’ meetings
Ticketing queue with supporters’ manual to track support cases
Introduction of ‘Federated cloud user support network’
18 teams,12 countries, 14 meetings
Target groups
RIs, Future and Emerging Technologies (FET) (besides the CCs!)
H2020 projects and structured communities – 11
Long-tail of science  brokered to national teams
SME and industry  by NA2
Popular topics
(Federated) community cloud
Containers, orchestrators
Data sharing, data access
‘Community-specific AAI’
Spread across disciplines, incl. CCs (D2.8 includes details):
Biological & medical (incl. biodiversity, ecosystem): 47.4% (18/38)
Earth sciences: 21% (8/38)
Physics (incl. astro, particle, laser, etc.): 21.2% (8/38)
Digital humanities (incl. languages): 5.2% (2/38)
Agriculture: 2.6% (1/38)
Nanotechnology: 2.6% (1/38)
SA2 Knowledge Commons

13. Support for new communities – Some examples
ICOS – Coupling compute with EUDAT storage
Euro-Argo – ‘Data subscription personal cloud’
ASTRON IBM research lab – network topology research on MPI clusters
SKA and LOFAR –adoption of cloud services …
BILS (Bioinformatics Infrastructure for Life Science)
PconsC2, Pcons, TOPCONS2 servers on Federated Cloud
TOPCONS2 predicted more than 2 million protein sequences by 5000 users from 69 countries 1 2 3 4 5 6 7 8
Outreach
Tech.supp
Reqs.
Co-devel.
Testing
Exploit.
SLA arran.
Operation
Human Brain Project
Access to brain images with Federated Open Data (JRA2)
10 TB dataset hosted at CYFRONET, DESY
Peachnote–Search inside music scores
Search engine hosted on Jüelich cloud site
Tens of thousands of users every day - through IMSLP, the largest sheet music library
SA2 Knowledge Commons

14. Support for new communities – Some examples
NEW CC in PY2
EMSO
Setup Hadoop clusters on EGI OpenStack sites with Sahara
Implemented on INFN-Bari cloud site
ICOS – Coupling compute with EUDAT storage
Euro-Argo – ‘Data subscription personal cloud’
ASTRON IBM research lab – network topology research on MPI clusters
SKA and LOFAR –adoption of cloud services …
BILS (Bioinformatics Infrastructure for Life Science)
PconsC2, Pcons, TOPCONS2 servers on Federated Cloud
TOPCONS2 predicted more than 2 million protein sequences by 5000 users from 69 countries 1 2 3 4 5 6 7 8
Outreach
Tech.supp
Reqs.
Co-devel.
Testing
Exploit.
SLA arran.
Operation
Human Brain Project
Access to brain images with Federated Open Data (JRA2)
10 TB dataset hosted at CYFRONET, DESY
Peachnote–Search inside music scores
Search engine hosted on Jüelich cloud site
Tens of thousands of users every day - through IMSLP, the largest sheet music library
SA2 Knowledge Commons

15. Distributed Competence Centre
Competence Centres: SA2.3-10
ESFRI Landmark
Distributed Competence Centre
BBMRI
DARIAH
EISCAT-3D
ELIXIR
EPOS
LifeWatch
MoBrain
Disaster mitigation
EMSO (with MoU)
NEW!
ESFRI Landmark
ESFRI Landmark
ESFRI Project
ESFRI Projects:
The ESFRI Projects have been selected for scientific excellence and maturity and are included in the Roadmap in order to underline their strategic importance for the European Research Infrastructure system and support their timely implementation. The ESFRI Projects can be at different stages of their preparation according to the date of inclusion in the ESFRI Roadmap.
ESFRI Landmarks:
The ESFRI Landmarks are the RIs that were implemented or started implementation under the ESFRI Roadmap and are now established as major elements of competitiveness of the European Research Area. The ESFRI Landmarks need continuous support for successful completion, operation and upgrade in line with the optimal management and maximum return on investment.
ESFRI Landmark
ESFRI Landmark
ESFRI Landmark
ESFRI Project
SA2 Knowledge Commons

16. Competence Centres: Diverse interests, different timelines
Number of relevant EGI service areas
Several, interlinked areas
EPOS
ELIXIR
MoBrain
AAI
Federated cloud
Federated Open Data
Application porting
Core platform (AAI, Service registry)
Federated Cloud
VM Marketplace
Application/tool porting
GPGPUs
Federated cloud
Community platform
Application porting
DARIAH
Disaster Mitigation
LifeWatch
Few, independent areas
DataHub
Real-time search application
Compute apps
Data-intensive computing
Federated cloud
Community portals
R and citizen tools
Federated cloud
Community VREs
RIs on the right are more strongly connected to NGIs, RIs in the left are much less
BBMRI
EISCAT_3D
Private clouds
Controlled data sharing
One/two areas
Data catalogue
HTC
Service maturity target (by end of EGI-Engage)
Pilot exploration of e-infra services
Production e-infra services
SA2 Knowledge Commons

17. CCs achievements in PY1 Implementation plans; Software tools/services
9 milestones and deliverables in PY1
Requirements and implementation plans for
Software package to deploy private cloud at a biobank (BBMRI)
Life science applications – 5 (ELIXIR)
Earth science use cases – 3 (EPOS)
Community application:
Image-based flora recognition (Pattern recognition) (LifeWatch)
R tools for ecosystem modelling (LifeWatch)
Data repository for digital humanities (DARIAH)
Real-time search and information retrieval (DARIAH)
User portals
EISCAT
MoBrain
SA2 Knowledge Commons

18. CCs in the Engagement pipeline
Requirement collection & system design
Co-development
Tests, evaluation
Dissemin./
Exploitation
SLAs, operation
ELIXIR
Compute Platform with AAI
5 use cases (M6.3)
Ongoing
BBMRI
Private clouds (M6.2)
Data workflows
MoBrain
Community portal
Done
Done (D6.4)
Started (ISGC, Utrecht, etc.)
SLA-OLA signed in Q1 2016
Cryo-EM in the cloud (SCIPION)
GPGPU application gateways
DARIAH
Data repository
Done (D6.2)
Real-time search
Done/Ongoing (D6.5)
Application gateway
LifeWatch
GPU & cloud infra. deployment
Apps. for citizen science
Done (D6.1)
EISCAT_3D
Data Portal
Done–Proof-of-concept (D6.3)
Data & Compute Portal
EPOS
3 use cases (M6.4)
Disaster Mitigat. ELIXIR
2 simulation gateways
gWRF and iCOMCOT gateways
3 core use cases
Typhoons (Haiyan, Morakot)
2014 rainfall in Malaysia
Thailand flood in 2011
2 external use cases (NE, VT)
WP6 Knowledge Commons

19. What have the PY1 outcomes enabled?
ELIXIR:
Implementation of ‘ELIXIR Compute Platform’; Deployment exists at 2 sites, ongoing at 3 more sites
BBMRI:
Software package for private biobank cloud installations (IaaS+PaaS); Deployments & workflows in PY2
MoBrain
Application portals to perform multi-scale simulations on mixed CPU and GPU systems
DARIAH
Demonstrators to capture the ‘cloud interest’ of digital humanities; Example use cases from the domain
LifeWatch
Reaching citizen scientists, with demonstrated use at Real Jardín Botánico; Consolidation of R tools
EISCAT_3D
Initial data schemas and data access & management services for EISCAT_3D
EPOS
An ecosystem with access control, data sharing, cloud, thematic exploitation platforms
Disaster Mitigation
Re-simulating typhoon events that caused 7,000 fatalities and 9 billion USD damage (in 2009, in 2013)
ELIXIR:
- Deployment finished at CESNET and GRNET; Ongoing at CSC, EBI, FRANCE
SA2 Knowledge Commons

20. Building the ‘Knowledge Commons’
People’s knowledge
Brought together approx. 70 experts from 50 institutes
EMSO joining as unfunded Competence Centre in PY2
21 training events (f2f and webinar)
International network of cloud user support teams
Technical knowledge
25 software service for/by competence centres
Each with user and technical documentations; Open licensed source code
Infrastructure knowledge
8 discipline-specific infrastructure Virtual Organisations
2 generic Virtual Organisations (training and fedcloud.egi.eu)
SA2 Knowledge Commons

21. Use of Resources and Problems
Numbers will be provided by PO, work package leaders must provide explanation

22. REPORTED in PY1 (% of total) EXPLANATION OF DIFFERENCE
Use of resources
TASK
TOTAL EFFORT M1-30 (PMs)
REPORTED in PY1 (% of total)
EXPLANATION OF DIFFERENCE
SA2.1 – Training
37.0
23%
Broadening activity in PY2-3
SA2.2 – Tech. User Support
56.5
41%
Uniform intensity
SA2.3 - ELIXIR M7-M24
36.5
13%
Started in M7 with requir. collection
SA2.4 - BBMRI M7-M30
29.0
34%
Started in M7
SA2.5 - MoBrain
59.5
59%
Parallel development activities in PY1
SA2.6 – DARIAH
62.5
51%
SA2.7 - LifeWatch
60.0
29%
Broadening activities in PY2-3
SA2.8 - EISCAT_3D
28.0
26%
Technology pathfinding activity in PY1
SA2.9 - EPOS
30.5
18%
Start with requirement collection.
SA2.10 – Disaster Mitigation
57.0 (unfunded)
Use case pathfinding activities in PY1
SA2 Knowledge Commons

23. Problems EISCAT_3D portal (D6.3)
Baseline portal technology had to be replaced
Mitigation:
External technology was brought in: DIRAC
Proof-of-concept portal setup released in March
Positive user evaluation  Effort reallocation for PY2
Adding computing capabilities in 2nd portal version: next 18 months
DARIAH Search and Information Retrieval application (D6.5)
Development took more time and effort than planned
Full version will be available in Q (Instead of end of Feb)
Unfunded extra effort by GWDG (Germany)
SA2 Knowledge Commons

24. Plan for PY2

25. Plans PY2 SA2.1: Training SA2.2: Technical user support
Emerging solutions: GPGPU platform, Federated Data Manager, DataHub, IdP proxy, long tail platform
CC services and demonstrators
EGI-EUDAT reference course
Further expansion of cloud training portfolio (PaaSs, Docker, APIs)
SA2.2: Technical user support
Follow-up active use cases – for example ‘Design your e-infrastructure workshop’
Active use of new platform for the long tail of science
Engage with additional communities
New opportunities in ESFRI roadmap 2016
New opportunities by H2020 projects – e.g. VREs, RI-related projects
SA2 Knowledge Commons

26. PY2 plans – Competence Centres
Requirement collection & system design
Co-development
Tests, evaluation
Dissemin./
Exploitation
SLAs, operation
ELIXIR
Compute Platform with AAI
5 use cases (M6.3)
Ongoing
BBMRI
Private clouds (M6.2)
Data workflows
MoBrain
Community portal
Done
Done (D6.4)
Started (ISGC, Utrecht, etc.)
SLA-OLA signed in Q1 2016
Cryo-EM in the cloud (SCIPION)
GPGPU application gateways
DARIAH
Data repository
Done (D6.2)
Real-time search
Done/Ongoing (D6.5)
Application gateway
LifeWatch
GPGPU, Cloud infra. Services
Apps. for citizen science
Done (D6.1)
EISCAT_3D
Data Portal
Done–Proof-of-concept (D6.3)
Data & Compute Portal
EPOS
3 use cases (M6.4)
Disaster Mitigat. ELIXIR
2 simulation gateways
gWRF and iCOMCOT gateways
3 core use cases
Typhoons (Haiyan, Morakot)
2014 rainfall in Malaysia
Thailand flood in 2011
2 external use cases (NE, VT)

27. PY2 plans – Competence Centres
Requirement collection & system design
Co-development
Tests, evaluation
Dissemin./
Exploitation
SLAs, operation
ELIXIR
Compute Platform with AAI 5 use cases (M6.3)
Operates in 5 ELIXIR nodes
Completed (D6.10, D6.15)
EXCELERATE
Jointly by ELIXIR and EGI
BBMRI
Private clouds (M6.2)
Operated in 3 biobanks
Completed (D6.11)
Common IT serv.
Data workflows (D6.8)
Ongoing
MoBrain
Community portal
Done
Done (D6.4)
SLA-OLA signed in Q1 2016
Cryo-EM in the cloud (SCIPION)
Done (D6.14)
GPGPU application gateways
Completed (D6.7, D6.12)
Completed
DARIAH
Data repository
Done (D6.2)
Real-time search
Done/Ongoing (D6.5)
Application gateway
Completed (D6.17
LifeWatch
GPGPU, Cloud infra. Services
GPGPU apps in production
Apps. for citizen science
Done (D6.1, D6.18)
EISCAT_3D
Data Portal
Done–Proof-of-concept (D6.3)
Data & Compute Portal
EPOS
3 use cases (M6.4)
Completed (D6.13, D6.19)
Ongoing(Compute)
Ongoing (Data)
Disaster Mitigat. ELIXIR
2 simulation gateways
gWRF and iCOMCOT gateways
Completed (D6.9)
3 core use cases
Typhoons (Haiyan, Morakot)
Completed (D6.20)
2014 rainfall in Malaysia
Thailand flood in 2011
2 external use cases (NE, VT)
Completed (Nepal, Vietnam)
SA2.3-10: 13 deliverables in next 12 months

28. Summary
Objective 1 (O1): The continued coordination of the EGI Community
Continued coordination of ‘EGI Engagement’ activity
Deeper collaborations: 27 RIs/FETs, 11 communities, numerous long-tail users
Nurtured the network of Federated Cloud user support teams
Objective 5 (O5): Promotion the adoption and extension of the current EGI services
Designed EGI-based community platforms for ELIXIR, BBMRI, EPOS
Launched co-developments of EGI-based services for 8 ESFRI communities
Supported/delivered 21 training events
Reached new users: from citizen science ecologists to HBP, EMSO, SKA
SA2 Knowledge Commons

29. Next: Jesús Marco de Lucas, IFCA-CSIC (Spain)
Coord. of LifeWatch Competence Center