Institute of Information and Communication Technologies – Bulgaria and IT Assoc. Prof. Emanouil Atanassov Head of Grid Technologies and Applications Department Institute of Information and Communication Technologies – BAS Email: emanouil@parallel.bas.bg

Historic perspective IICT-BAS operated different types of clusters and supercomputers like the Origin 2000 IICT-BAS coordinates the Bulgarian participation in the European Grid Initiative, first through the EGEE series of projects, then EGI-InSPIRE and EGI-Engage IICT-BAS is one of the leading partners in a series of projects that spread the expertise of advanced countries in the region to the less developed countries Innovative scientific applications are developed and run on the available HPC equipment at IICT-BAS by teams of scientists from institutes of the Bulgarian Academy of Sciences and leading universities in Bulgaria. Clusters at IM-BAS, INRNE-BAS and Sofia Universtity were added to the EGEE production infrastructure during EGEE and EGEE 2 projects.

Regional Cooperation -History PROJECTS

Regional partnerships – from infrastructure to applications

National Roadmap for Research Infrastructures RI: National Center for HPC and Distributed Computing - NC4HPC&DC Financial coordinator: Ministry of Education and Science; Ministry of Transport, Information Technology and Communications. Scientific Coordinator: Institute of Information and Communication Technologies - Bulgarian Academy of Sciences (IICT-BAS) Consortium for supercomputers applications (since 2009 г.) Consortium for distributed (Grid and Cloud) applications (set up - 2004, updated - 2012) Sofia University “St Kliment Ohridski” Institute for Nuclear Research and Nuclear Energy - Bulgarian Academy of Sciences (INRNE-BAS) Technical University – Sofia (TU-Sofia Medical University – Sofia (MU-Sofia) Technical University - Gabrovo National Institute of Geophysics, Geodesy and Geography -Bulgarian Academy of Sciences (NIGGG-BAS) Institute of Molecular Biology “Roumen Tsanev” -Bulgarian Academy of Sciences (IMB-BAS) Institute of Organic Chemistry with Centre for Phytochemistry - Bulgaria Academy of Sciences ( IOCCP-BAS) Institute of Mechanics – Bulgaria Academy of Sciences (IM-BAS) Remark: The consortium for supercomputer applications closely cooperates with Association “National Centre for Supercomputing Applications” (NCSA), which represents Bulgaria in PRACE Institute of mathematics and Informatics – Bulgarian Academy of Sciences (IMI – BAS)

HPC resources in Bulgaria

HPC Center “Avitohol” at IICT 150 HP Cluster Platform SL250S GEN8 servers with 2 Intel Xeon E 2650 v2 CPUs and 2 Intel Xeon Phi 7120P coprocessors Site IICT-BAS/Avitohol Manufacturer Hewlett-Packard Cores 20700 Interconnection FDR InfiniBand Theoretical Peak Performance 412.3 Tflop/s RMAX Performance 264.0 Tflop/s Memory 9600 GB Operation System Red Hat Enterprise Linux for HPC Storage capacity 96 TB SAN Top500 List on 389 place (Nov 2015) http://www.top500.org/system/178609 tt Users workstations 10 Gbps

The Supercomputer System Avitohol at IICT-BAS 8 water-cooled racks of type HP MCS 200, paired in couples. Each pair provides power and cooling for up to 50 kW of equipment About 90% of the computational power comes from the accelerators – one 7120P coprocessor achieves 1.25 TFlop/s in double precision. Total energy use at maximum load ~ 250kW.

The Supercomputer System Avitohol at IICT-BAS RedHat Enterprise for HPC Intel Cluster Studio (compilers, MPI,MKL) Moab for resource management with web and command line interface High-end database software - SAP HANA – in-memory database with advanced real-time analytics capabilities

The Supercomputer System Avitohol at IICT-BAS 150 dual-socket nodes HP SL250s Gen8 with 2 Intel Xeon E5-CPU E5-2650 v2 @ 2.60GHz and 2 Intel Xeon Phi 7120P coprocessors, 64 GB RAM Interconnected with fully non-blocking FDR InfiniBand Lustre parallel filesystem, 96 TB of disk storage

The Supercomputer System Avitohol at IICT-BAS Linpack tests: 222.0 TFlop/s achieved in June, the system entered Top500 in 332nd place. Further tuning of hardware and software produced the new result of 264 TFlop/s, maintaining presence in the November Top500 list. For comparison the Blue Gene/P supercomputer achieved 23.9 TFlop/s and was last present in Top500 list in November 2009 in 379th place.

National projects supported Project Name Institution PI Name Grant Development and Investigation of quasi-Monte Carlo Algorithms for Extreme Parallel Computer Systems, funded by Bulgarian NSF, 2014 – 2016 IICT-BAS Assoc. Prof. T. Gurov #DFNI-I02/8 Estimate of brown bear population in Bulgaria on the basis of mathematical, statistical and biological analysis of monitoring data”, funded by PUDOOS, 2013 – 2015 #D9190 Parallel and Distributed Computing Practices Assoc. Prof. I. Lirkov Numerical methods for coupled systems and computer modelling of biomedcal and environmental problems Assoc. Prof. K. Georgiev #DFNI-I01/5 Efficient Parallel Algorithms for Large-Scale Computational Problems, 2015 – 2016 Prof. I. Dimov #DFNI-I02/20 Preparation of operational numerical physical model of the Black Sea Faculty of Physics-SU Assoc. Prof. E. Peneva Dynamics , mechanisms and sequence of bonding of the proteins responsible for DNA repair in living cells IMB-BAS Assoc. Prof. S. Stoynov #DFNI-Б02/16 Modeling of catalytic systems Faculty of Chemistry-SU Prof. G. Vaysilov Efficient numerical methods and parallel algorithms for dynamical analysis of plate structures, 2016 – 2017 Assist. Prof. PhD S. Stoykov 229/18.02.2016 New mathematical methods of machine learning with applications in modern technology for genetic sequencing IMI-BAS Prof. Е. Stoimenova #DFNI-I02/19 Fundamental research on stochastic branching processes – borderline behavior, statistical inferences and applications FMI-SU Prof. M. Bojkova #DFNI-I02

International projects supported Acronym Project Name PI Name Grant VI-SEEM VRE for regional Interdisciplinary communities in Southeast Europe and the Eastern Mediterranean, 2015 – 2018 Assoc. Prof. T. Gurov #675121, EC H2020 MMAC Centre of Excellence for Mathematical Modeling and Advanced Computing, 2015 – 2016 Corr. Mem. S. Margenov #664406, EC H2020 SESAME-NET Supercomputing Expertise for SmAll and Medium Enterprise Network, 2015 – 2017 #654416, EC H2020 EGI-Engage Engaging the EGI Community towards an Open Science Commons, 2015 – 2017 Assoc. Prof. E. Atanassov #654142, EC H2020 CRoNoS Computationally-intensive methods for the robust analysis of non-standard data, 2015 – 2019 Prof. A. Karaivanova COST Action: IC1408 PRACE 4 IP PRACE Fourth Implementation Phase, 2015 – 2017 Prof. S. Markov #653838, EC H2020 AComIn Advanced Computing for Innovation, 2012 – 2015 Prof. G. Angelova #316087, EC FP7 QTLeap Quality Translation by Deep Language Engineering Approaches, 2013 – 2016 Assoc. Prof. K. Simov #610516, EC FP7

Extension HPCG cluster at IICT-BAS – BG01-IPP 36 blade servers HP BL 280c, deployed in 3 HP Cluster Platform Express 7000 enclosures, each with 2 Xeon X5560 @ 2.80GHz, 24 GB RAM – 576 cores total with more than 3 Tflops peak performance; 8 dual-socket HP DL 380 G6 with dual Intel X5560 @ 2.8 Ghz, 32 GB RAM. Non-blocking Infiniband interconnection @ 20 Gbps, 92% efficiency on LINPACK - 3 TFlops Total disk storage more than 132 TB from three disk systems, interconnected with Fibre Channel. 2 HP ProLiant SL390s G7 4U servers with 16 NVIDIA Tesla M2090 graphic cards (total 8192 GPU cores with 10.64 TFlops in double precision); HP SL270s Gen8 4U server with 8 Intel Xeon Phi 5110P Coprocessors (total 480 cores, 1920 threads, 8.088 TFlops of double-precision peak performance per server). Total peak theoretical peak performance 22.93 TFlops

Organization of access Bulgarian scientists can obtain access to the HPC systems of IICT-BAS by filling request forms that are assessed on basis of scientific merit. Remote access through ssh: gw.avitohol.acad.bg Batch system: torque/moab Development tools: Intel Parallel Studio XE 2016 GNU Compiler Collection MVAPICH 2.2 Open MPI Support helpdesk: avitohol-support@parallel.bas.bg Access policies: http://www.hpc.acad.bg/access-policies/ More information at: http://www.hpc.acad.bg/

Organization of access IICT-BAS maintains the BG.ACAD CA certification authority, which issues x509 certificates for use in Grids and PRACE. BG.ACAD CA is member of EUGRID PMA organization, thus Bulgarian certificates are accepted in all European resources and the collaborating Asian-Pacific and American resources. Avitohol is reachable through the PRACE internal network

Software and tools Libraries: MKL FFTW PETSc SLEPc GSL NetCDF … Languages, tools: R, Python, Java cmake, gnuplot Paraview Scalasca gdb, gprof

IICT-BAS is the Bulgarian representative in the European Grid Initiative - EGI Data Providers Technology Providers Service Providers All

European Grid Initiative - EGI Federated services BBMRI DARIAH EISCAT-3D ELIXIR EPOS LifeWatch MoBRAIN/ INSTRUCT Environment - disaster mitigation Innovating with RIs - The Distributed Competence Centre of EGI-Engage

Regional HPC and distributed computing initiative – VI-SEEM The project VI-SEEM aims to integrate all kinds of electronic infrastructure in the region – Grid, Cloud, HPC, BigData, in order to provide single point of access for scientists in the region of South Eastern Europe and the Mediterranean. Bulgaria is the leader in the HPC area. Provides user-friendly integrated e-Infrastructure platform for Scientific Communities in: Climatology, Life Sciences, Cultural Heritage for the SEEM region by linking compute, data, and visualization resources, as well as services, software and tools. Avitohol is the most powerful supercomputer in VI-SEEM

VI-SEEM Geographical and historical context VRE for regional Interdisciplinary communities in SEE and EM Merging of SEE and EM regions SEE: network SEEREN1-2, Grid SEE-GRID-1/2/SCI, HPC HP-SEE EM: HPC LinkSCEEM1-2 Participant no. Participant organisation name Part. short name Country 1 (Coord) GREEK RESEARCH AND TECHNOLOGY NETWORK S.A. GRNET Greece 2 THE CYPRUS INSTITUTE CyI Cyprus 3 INSTITUTE OF INFORMATION AND COMMUNICATION TECHNOLOGIES – BULGARIAN ACADEMY OF SCIENCES IICT-BAS Bulgaria 4 INSTITUTE OF PHYSICS BELGRADE IPB Serbia 5 NATIONAL INFORMATION INFRASTRUCTURE DEVELOPMENT INSTITUTE NIIF Hungary 6 WEST UNIVERSITY OF TIMISOARA UVT Romania 7 POLYTECHNIC UNIVERSITY OF TIRANA UPT Albania 8 UNIVERSITY OF BANJA LUKA UNI BL Bosnia and Herzegovina 9 SS CYRIL AND METHODIUS UNIVERSITY OF SKOPJE UKIM FYR of Macedonia 10 UNIVERSITY OF MONTENEGRO UOM Montenegro 11 RESEARCH AND EDUCATIONAL NETWORKING ASSOCIATION OF MOLDOVA RENAM Moldova 12 INSTITUTE FOR INFORMATICS AND AUTOMATION PROBLEMS OF THE NATIONAL ACADEMY OF SCIENCES OF THE REPUBLIC OF ARMENIA IIAP-NAS-RA Armenia 13 GEORGIAN RESEARCH AND EDUCATIONAL NETWORKING ASSOCIATION GRENA Georgia 14 BIBLIOTHECA ALEXANDRINA BA Egypt 15 INTER UNIVERSITY COMPUTATION CENTER IUCC Israel 16 SYNCHROTRON-LIGHT FOR EXPERIMENTAL SCIENCE AND APPLICATIONS IN THE MIDDLE EAST SESAME Jordan

Regional HPC and BigData Resources Main resource providers – Bulgaria, Greece, Serbia. Heterogeneous computing resources – CPU-only, GPGPU-enmabled, Xeon Phi, IBM Cell Bulgaria is the leader in computing. Need for larger and better storage facilities

VI-SEEM technology context Overall objective: Provide user-friendly integrated e-Infrastructure platform for Scientific Communities in Climatology, Life Sciences, and Digital Cultural Heritage for the SEEM region; by linking compute, data, and visualization resources, as well as services, software and tools. Diverse computing technologies Advent of big data Service orientation

EU Network of HPC Competency Centres working with SMEs – SESAME NET 14 Partners , 13 Member States ARCTUR, Slovenia (coordinator) Bangor University, United Kingdom Irish High End Computing, National University of Galway, Ireland Fraunhofer Institute for Algorithms and Scientific Computing (SCAI), Germany Poznan Supercomputing and Networking Center IBCH (Inst. of Bioorganic Chemistry) PAS, Poland Greek Research and Technology Network S.A., Greece Centro de Supercomputación de Galicia CESGA, Spain IT4Innovations National Supercomputing Center, VŠB – Techn. University of Ostrava, Czech Republic Yotta Advanced Computing d.o.o., Croatia West University of Timisoara UVT, Romania Inst. of Inform. and Comm. Technologies of BAS (IICT-BAS), Bulgaria Ruder Bošković Institute, Croatia Herculesstichting / Flemish Supercomputing Centre, Belgium Vilnius University – Faculty of Mathematics and Informatics, Lithuania

BG05-SUGRID perspectives The BG05-SUGrid Grid site is currently the second Bulgarian production Grid Site. Its current hardware is rather limited, however substantial upgrade is planned: Computing - 72 servers with dual Intel L5640 (12 cores)  - 864 total physical cores, with capability for hyperthreading Storage - 15 servers with 24*2TB hdd each – 720 TB total

Conclusions The Avitohol supercomputer is the top HPC system in Bulgaria and ensures the needs of Bulgarian scientists in the near term. It is used mostly for HPC tasks, but some distributed computing research is also performed on it. Grid clusters like BG01-IPP and BG05-SUGRID support diverse virtual organizations, including those from CERN. Advanced hardware sustains involvement in both infrastructure provision and development activities The equipment and software is upgraded regularly, following the best practices in the IT Ensuring governmental support for the upgrade and maintenance of the infrastructure is crucial for its capability to support advanced research