1. A Practical Evaluation of Hypervisor Overheads Matthew Cawood Supervised by: Dr. Simon Winberg University of Cape Town Performance Analysis of Virtualization for High Performance Computing

2. Matthew Cawood (UCT) Overview 1.Background 2.Research Objectives 3.HPC 4.Virtualization 5.Performance Tuning 6.The Research Cluster 7.Benchmark Selection 8.Results 9.Conclusions

3. Matthew Cawood (UCT) 1. Background BSc (Eng) final year research project Based in CHPC’s Advanced Computer Engineering (ACE) Lab Access to research cluster currently being commissioned Project focused on evaluating cluster hardware and software

4. Matthew Cawood (UCT) 2. Research Objectives 1.Present an in-depth report on the current technologies being developed in the field of High Performance Computing. 2.Provide a quantitative performance analysis of the costs associated with Virtualization, specifically in the field of HPC.

5. Matthew Cawood (UCT) 3. High Performance Computing HPC data centres are rapidly growing in size and complexity Current emphasis placed on improving efficiency and utilization Wide selection of applications/requirements Bioinformatics Astrophysics Simulation Modelling

6. Matthew Cawood (UCT) 4. Virtualization

7. Matthew Cawood (UCT) 4. Virtualization

8. Matthew Cawood (UCT) 4. Virtualization

9. Matthew Cawood (UCT) 4. Virtualization

10. Matthew Cawood (UCT) 4. Virtualization

11. Matthew Cawood (UCT) 4. Virtualization

12. Matthew Cawood (UCT) 5. Performance Optimizations Host memory reservation of Linux huge pages KVM vCPU pinning to improve NUMA cell awareness

13. Matthew Cawood (UCT)

14. 6. The Research Cluster Compute Nodes: 2x Intel Xeon E5-2690, 20MB L3 cache, 2.90 GHz 256GB, DDR3-1600, CL11 Mellanox ConnectX-3 VPI FDR 56Gbps HCA Gigabit Ethernet NIC Switch Infrastructure: Mellanox SX6036 FDR 36 port Infiniband Switch

15. Matthew Cawood (UCT) 6. The Research Cluster CentOS 6.4 OFED 2.0 (with SR-IOV) OpenNebula 4.2

16. Matthew Cawood (UCT) 7. Performance Benchmarks HPC Challenge HPLinpack MPI Random Access STREAM Effective bandwidth & latency OpenFOAM 7 million cell, 5 millisecond transient simulation snappyHexMesh

17. Matthew Cawood (UCT) 8. Results

18. Matthew Cawood (UCT) 8.1 Software Comparison HPLinpack throughput comparison of compiler selection

19. Matthew Cawood (UCT) 8.2 Single Node Evaluation HPLinpack throughput efficiency of virtual machines MPI Random Access Performance STREAM Memory Bandwidth

20. Matthew Cawood (UCT) 8.3 Cluster Evaluation HPLinpack throughput efficiency of virtual machines

21. Matthew Cawood (UCT) 8.3 Cluster Evaluation OpenFOAM runtime efficiency of virtual machines

22. Matthew Cawood (UCT) 8.4 Interconnect Evaluation Typical Verbs Latency of virtual machinesTypical IPoIB Latency of virtual machines Native Verbs Vs. IP over Infiniband

23. Matthew Cawood (UCT) 8.5 Supplementary Tests Intel ® Hyper-threading HPLinpack throughput

24. Matthew Cawood (UCT) 9. Conclusions KVM provides good performance for HPC Tuning is necessary to further improve performance Efficiency is highly application dependant SR-IOV for Infiniband effectively reduced I/O Virtualization overheads Synthetic and real-world results often contradict

25. Matthew Cawood (UCT) Questions ?