1. PL-Grid: Polish Infrastructure for Supporting Computational Science in the European Research Space 1 Distributed Computing Instrastructure as a Tool for e-Science ACK Cyfronet AGH PL-Grid Consortium Jacek Kitowski, Kazimierz Wiatr, Łukasz Dutka, Maciej Twardy, Tomasz Szepieniec, Mariusz Sterzel, Renata Słota and Robert Pająk PPAM 2015, 7-9.09.2015, Kraków

2. 2 Outline National e-Infrastructure Assumptions and foundations Tool for e-Science e-Infrastructure creation – motivation, background and issues Conceptualization and implementation PLGrid case study Enhancement of Achievements Platforms and Environments – Selected Examples Conclusions

3. 3 e-Infrastructure Creation Motivation and Background Experiments in silico: Computing and Data for Open Science International collaboration User/platform driven e-infrastructure innovation (e-Science and e-Infrastructure interaction) Increasing importance of Computational Science and Big Data Analysis 1st: Theory 2nd: Experiment 4th Paradigm: Data Intensive Scientific Discovery 3rd: Simulation Data intensive computing Numerically intensive computing Computational Science problems: Algorithms, environments and deployment Future and emerging technologies 4 th paradigm, distributed, grid and cloud computing, Data Farming Needs: Preventing users from technical problems Expert support for making science Increase of resources Openess for future paradigms Activity initiated by Cyfronet

4. 4 Organizational: horizontal perspective - federation of computer centres supporting the e-infrastructure with different kinds of resources and competences vertical perspective - involvement of computer, computational and domain-specific experts into e-infrastructure operations Meeting user demands in the field of grand challenges applications Activity supported by users with scientific achievements and by well-defined requirements Meeting user demands Organizational Technological Energy awareness Technological: different computing hardware and software various middleware environments Energy awareness: optimal scheduling strategies of computing jobs among federation resources to minimize energy consumption as a whole Synergistic effort in several dimensions: e-Infrastructure Creation Issues

5. 5 PL-Grid Consortium Consortium creation – 2007 a response to requirements from Polish scientists due to ongoing eScience activities in Europe and in the World Aim: significant extension of computing resources and solutions provided to the scientific community PL-Grid Programme Development based on (SWOT analysis): projects funded by the European Regional Development Fund as part of the Innovative Economy Program close international collaboration (EGI, ….) previous projects (5FP, 6FP, 7FP, EDA…) National Network Infrastructure available: Pionier National Project computing resources: Top500 list Polish scientific communities: ~75% highly rated Polish publications in 5 Communities PL-Grid Consortium members: 5 High Performance Computing Polish Centres, representing the Communities coordinated by ACC Cyfronet AGH

6. ACK Cyfronet AGH 42 years of expertise High Performace Computing High Performance Networking Centre of Competence Human Resources Infrastructure Resources Network Resources Rank TOP500 SiteSystemCores R max Tflops R peak Tflops 49 VII.2015 Cyfronet Poland Prometheus HP Apollo 8000 41,4721262.41658.9 269 VII.2015 Cyfronet Poland Zeus Cluster Platform Hewlett-Packard 25,468266.9373.9 Social Networking

7. 7 The most powerful HPC Asset (in Poland) Prometheus Cluster (2014/2015) R peak = 1658.9 TFlops 1728 servers 41,472 Haswell cores 216 TB RAM (DDR4) 10 PB disks, 180 GB/s HP Apollo 8000 In operation April 2015 49th position on the July 2015 edition of the TOP500 list Q4 2015 Extensions R peak = 483.8 TFlops 504 servers 12,096 Haswell cores R peakNVIDIA = 256.3 TFlops 144 Nvidia K40 XL In SUMMARY: 2,4 PFlops (with GPU)

8. 88 TOP500, July 2015 Polish Sites RankSiteSystemCores Rmax (TFlop/s) Rpeak (TFlop/s) 49 269 Cyfronet, Krakow Prometheus - HP Apollo 8000, Xeon E5- 2680v3 12C 2.5GHz, Infiniband FDR Hewlett-Packard Zeus - Cluster Platform SL390/BL2x220, Xeon X5650 6C, 2.660GHz, Infiniband QDR, NVIDIA 2090 Hewlett-Packard 41,472 25,468 1,262.4 266.9 1,658.9 373.9 126 TASK, Gdańsk Tryton - HP ProLiant XL230a Gen9, Xeon E5-2670v3 12C 2.3GHz, Infiniband Megatel/Action 17,280530.5635.9 135 WCSS, Wrocław BEM - Actina Solar 820 S6, Xeon E5- 2670v3 12C 2.3GHz, Infiniband FDR ACTION 17,280480.1635.9 155 NCNR, Świerk Świerk Computing Centre - Supermicro TwinBlade SBI-7227R/Bull DLC B720, Intel Xeon E5-2680v2/E5-2650 v3 10C 2.8GHz, Infiniband QDR/FDR Format, Bull, Atos Group 17,960423.2490.4 380 418 NGSC & ICM, University of Warsaw ORION - Dell PowerEdge R730, Xeon E5- 2680v3 12C 2.5GHz, Infiniband FDR, AMD FirePro S9150 Dell BlueGene/Q, Power BQC 16C 1.600GHz, Custom Interconnect IBM 16,800 16,384 198.8 189.0 903.0 209.7

9. 9 PLGrid PLUS (2011–2015) Number of people involved: ca. 120 Outcome: Focus on users (training, helpdesk…) Domain specific solutions: 13 domains (Specific computing environments) Extension of resources and services by: 500 Tflops, 4.4 PB Family of PL-Grid Projects coordinated by Cyfronet PL-Grid (2009–2012) Number of people involved: ca. 80 (total, from different Polish Centres) Outcome: Common base infrastructure National Grid Infrastructure (NGI_PL) Resources: 230 Tflops, 3.6 PB PLGrid NG (2014–2015) Expected outcome: Optimization of resources usage, training Extension of domain specific solutions by 14 add’l domains Extension of resources and services by: ca. 8 Tflops, some PB PLGrid CORE (2014–2015) (Cyfronet only) Expected outcome: Competence Center End-user services Open Science paradigm Large workflow applications Data Farming mass computation Extension of resources and services by: ca. 1500 Tflops, 25 PB

10. 10 Family of PL-Grid Projects coordinated by Cyfronet PLGrid PLUS (2011–2015) Outcome: Focus on users (training, helpdesk…) Domain specific solutions: 13 PL-Grid (2009–2012) Outcome: Common base infrastructure PLGrid NG (2014–2015) Outcome: Optimization of resources usage, training Extension of domain specific by 14 PLGrid CORE (2014–2015) Outcome: Competence Center Open Science paradigm (large workflow app., data farming mass comp., ……) End-user services 230 Tflops +1500 Tflops +500 Tflops +8 Tflops Assumed Performance Real Users

11. 11 PLGrid PLUS (2011–2015) Number of people involved: ca. 120 Outcome: Focus on users (training, helpdesk…) Domain specific solutions: 13 domains (Specific computing environments) Extension of resources and services by: 500 Tflops, 4.4 PB Family of PL-Grid Projects coordinated by Cyfronet PL-Grid (2009–2012) Number of people involved: ca. 80 (total, from different Polish Centres) Outcome: Common base infrastructure National Grid Infrastructure (NGI_PL) Resources: 230 Tflops, 3.6 PB PLGrid NG (2014–2015) Expected outcome: Optimization of resources usage, training Extension of domain specific solutions by 14 add’l domains Extension of resources and services by: ca. 8 Tflops, some PB PLGrid CORE (2014–2015) (Cyfronet only) Expected outcome: Competence Center End-user services Open Science paradigm Large workflow applications Data Farming mass computation Extension of resources and services by: ca. 1500 Tflops, 25 PB Real Users

12. 12 Summary of Projects Results (up-to-date) Close collaboration between Partners and research communities Development of tools, environments and middleware services, Clouds Integration, HPC, Data intensive, Instruments Development of 27 domain specific solutions Development of IT PL-Grid Infrastructure and ecosystem

13. 13 Summary of Projects Results (up-to-date) Facilitation of community participation in international collaboration EGI Council, EGI Executive Board FP7 (VPH-Share, VirtROLL….) EGI-InSPIRE, FedSM, … EGI-Engage, Indico DataCloud, EPOS, CTA, PRACE, H2020…. Publications 26 papers on PL-Grid Project results 36 papers on PLGrid Plus Project results 147 authors, 76 reviewers

14. 14 Journal Publications (subjective selection) JournalIF J.Chem.Theor.Phys.Appl.5.31 Phys.Lett. B6,019 J.High Energy Phys.6,22 Astonomy &Astrophys.4,479 Inorganic Chem.4,794 J.Org.Chem.4,638 Optic Lett.3,179 Appl.Phys.Lett.3.515 J.Comput.Chem.3,601 J.Phys.Chem. B3,377 Soft Matter4,151 Int.J.Hydrogen Energy2,93 Physica B1,133 JournalIF J.Chem.Phys.3,122 J.Phys.Chem.Lett.6,687 Phys.Chem.Chem.Phys.4,638 Fuel Processing Techn.3,019 J.Magn. & Magn. Mat.2,002 Eur.J.Inorg.Chem.2,965 Chem.Phys.Lett.1,991 Phys.Rev.B3,664 Eur.Phys.J.2,421 Future Gen.Comp.Syst.2,639 J.Phys.Chem. C4,835 Crystal Growth & Desing4,558 JournalIF Macromolecules5,927 Astrophys.J.Lett.5,602 Phys.Rev.Letters7,728 J.Chem.Theor.Appl.5,31 Astrophys.J6,28 Chem.Physics2,028 Molec.Pharmaceutics4,787 Eur.J.Pharmacology2,684 Energy4,159 Carbon6,16 J.Biogeography4,969 Electrochem.Comm.4,287 J.Magn.&Magn.Mat.1,892 Conferences: Cracow Grid Workshop (since 2001) KU KDM (since 2008)

15. 15 Summary of Projects Results (up-to-date) # users # users’ grants (active)

16. 16 Summary of Projects Results (up-to-date) Examples of active grants PROTMD (18.9.2015-18.9.2016) – Cyfronet Research on proteins using MD 25 mln hours (2,800 cores) PCJ2015GA (26.8.2015-31.12.2015) – ICM Research on connectome of nematodes using GA 15 mln hours (6,000 cores) PSB (1.3.2015-1.3.2016) – TASK, Cyfronet, ICMM, WCSS New characteristics of DNA in the context of tumor therapy 11 mln hours (1,200 cores)

17. 17 Summary of Projects Results (up-to-date)

18. 18 Deployed PLGrid IT Platforms and Tools – selected examples (by Cyfronet)

19. 19 GridSpace A platform for e-Science applications Experiment: an e-science application composed of code fragments (snippets), expressed in either general-purpose scripting programming languages, domain-specific languages or purpose-specific notations. Each snippet is evaluated by a corresponding interpreter. GridSpace2 Experiment Workbench: a web application - an entry point to GridSpace2. It facilitates exploratory development, execution and management of e-science experiments. Embedded Experiment: a published experiment embedded in a web site. GridSpace2 Core: a Java library providing an API for development, storage, management and execution of experiments. Records all available interpreters and their installations on the underlying computational resources. Computational Resources: servers, clusters, grids, clouds and e-infrastructures where the experiments are computed. Contact: E. Ciepiela, D. Harężlak, M. Bubak

20. 20 InSilicoLab science gateway framework Goals Complex computations done in non-complex way Separating users from the concept of jobs and the infrastructure Modelling the computation scenarios in an intuitive way Different granularity of the computations Interactive nature of applications Dependencies between applications Summary The framework proved to be an easy way to integrate new domain-specific scenarios Even if done by external teams Natively supports multiple types of computational resources Including private resources – e.g. private clouds Supports various types of computations Architecture of the InSilicoLab framework: Domain Layer Mediation Layer with its Core Services Resource Layer with different kinds of workers Different kinds of users  different kinds of resources Contact: J. Kocot, M. Sterzel, T. Szepieniec

21. 21 DataNet collaborative metadata management Objectives Provide means for ad-hoc metadata model creation and deployment of corresponding storage facilities Create a research space for metadata model exchange and discovery with associated data repositories with access restrictions in place Support different types of storage sites and data transfer protocols Support the exploratory paradigm by making the models evolve together with data Architecture Web Interface is used by users to create, extend and discover metadata models Model repositories are deployed in the PaaS Cloud layer for scalable and reliable access from computing nodes through REST interfaces Data items from Storage Sites are linked from the model repositories Contact: E. Ciepiela, D. Harężlak, M. Bubak

22. 22 Onedata transparent access to data A system that provides a unified and efficient access to data stored in organizationally distributed environments. Provides a uniform and coherent view on all data stored on the storage systems distributed across the infrastructure Supports working in groups by creation of an easy-to-use shared workspace for each group. Serves data efficiently Onedata Global Registry Contact: Ł. Dutka

23. 23 Scalarm data farming experiments Self-scalable platform for parametric studies Adapting to experiment size and simulation type Exploratory approach for conducting experiments Supporting online analysis of experiment partial results Integrates with clusters, Grids, Clouds Data farming experiments with an exploratory approach What problems are addressed with Scalarm ? Scalarm overview Scalarm Graphical User Interface Parameter space generation with support of design of experiment methods Accessing heterogeneous computational infrastructure 75% all submitted tasks Self-scalability of the management/execution parts Contact: R. Słota

24. 24 Rimrock access to resources A service which simplifies the management of processes and tasks executed in the PLGrid infrastructure. Contact: D. Harężlak Rimrock architecture simplicity – non-complicated integration with other applications, scripts and services interactivity – a user can modify working processes based on indirect results universalism – supported by many programming languages versatility – it allows to execute an application in a batch mode or start an interactive application user friendliness – it does not require advanced knowledge (basic information about Bash shell and curl command are sufficient to start using it) Rimrock features

25. 25 Cloud Computing The Cloud increases elasticity of research, as scientists can tune the virtual machines to their specific needs. The catalogue of VMs offered by PL-Grid contains many OSs. Cloud platform is also the best and in many cases the only solution for running jobs with legacy software packages. Open Nebula  migration to Open Stack, …. Cloud Platform for VPH-Share applications (Atmoshere env.) IaaS, PaaS, STaaS…. Contact: J. Meizner, T. Szepieniec, M. Radecki

26. 26 Cloud environment for VPH-Share app. Portal and Atmosphere

27. 27 Applications Catalog service Objective: to present in one place and in a uniform manner the current offer of the software available in the PLGrid infrastructure, broken down into supercomputing centers, clusters as well as categories and areas of application. Applications Catalog is a system collecting and providing information on the applications, development tools and libraries offered in the PLGrid infrastructure. It allows to search for applications, check the status of their operation, obtain information about changes and updates, as well as it provides documentation and examples of usage. It is designed for all those interested in the use of the applications available in the PLGrid infrastructure.

28. 28 Map-Reduce service Apache Spark 1.5.0 functionality: API, RDD, DataFrame, SQL Backend Execution: DataFrame and SQL Integrations: Data Sources, Hive, Hadoop, Mesos and Cluster Management R Language Machine Learning and Advanced Analytics Spark Streaming

29. 29 Summary and Conclusions Three dimensions of development: HPC/GRID/CLOUDs Data & Knowledge layer Network & Future Internet Deployments have the national scope; however with close European links Development oriented on end-users & research projects Achieving synergy between research projects and e-infrastructures by close cooperation and offering relevant services Durability at least 5 years after finishing the projects - confirmed in contracts Future plans: continuation of development Center of Excellence CGW, KUKDM as places to exchange experience and for collaboration between eScience centers in Europe

30. 30 More information Please visit our Web pages: http://www.plgrid.pl/en http://www.plgrid.pl CREDITS!

31. 31 Credits ACC Cyfronet AGH Michał Turała Marian Bubak Krzysztof Zieliński Karol Krawentek Agnieszka Szymańska Maciej Twardy Angelika Zaleska-Walterbach Andrzej Oziębło Zofia Mosurska Marcin Radecki Renata Słota Tomasz Gubała Darin Nikolow Aleksandra Pałuk Patryk Lasoń Marek Magryś Łukasz Flis ICM Marek Niezgódka Piotr Bała Maciej Filocha PCSS Maciej Stroiński Norbert Meyer Krzysztof Kurowski Tomasz Piontek Paweł Wolniewicz WCSS Jacek Oko Józef Janyszek Mateusz Tykierko Paweł Dziekoński Bartłomiej Balcerek TASK Rafał Tylman Mścislaw Nakonieczny Jarosław Rybicki … and many others….. Special thanks to many domain experts !