1. Dr Mark Parsons Commercial Director, EPCC m.parsons@epcc.ed.ac.uk +44 131 650 5022 Advanced Engineering Simulation and its future at EPCC Mark Parsons, EPCC

2. Outline Introduction to the EPCC Industry Hub Recent advanced engineering simulation projects Challenges facing simulation in the next decade eDIKT 2010

3. EPCC in 2010 Supercomputing Centre at The University of Edinburgh 20 years old 75 staff – highly experienced – wide range of skills Multidisciplinary Multi-funded – turnover £4.5 million – 95% from external sources – 50% of turnover involves industry Provides National HPC services –HECToR is hosted at Advanced Computing Facility eDIKT 2010

4. EPCC Industry Hub Cycles, Software and Skills One stop shop for industry supercomputing Unique range of services –Cycles – provision of compute and data resources –Software – provision of simulation and bespoke applications to meet industry challenges –Skills – understanding industry problems and finding solutions Benefits –Scotland – supporting local industry – creating jobs and growth –Inward investment – attracting companies eg. Xilinx, Cray –International leadership – worldwide research collaborations –Creating an important facility where we work together with end-user companies and leading HPC hardware and software vendors eDIKT 2010

5. HPC Adopter Programme Too few companies benefit from modelling and simulation Big hurdles to overcome for the new HPC user –Even for high-tech companies HPC Adopter Programme is designed to tackle this –Initial small 2-4 month projects designed to introduce companies to benefits of HPC –Initial access to HECToR and other HPC systems donated by EPCC –Companies pay for staff effort only –For Scottish companies Scottish Enterprise may also support staff costs through State Aid to company –To date SE have supported 3 HPC Adopter Programme pilots –Other companies have already benefitted from programme Long term virtuous circle of repeat business eDIKT 2010

6. Deep Casing Tools – Turbocaser Deep Casing Tools is a privately owned SME in Aberdeen. Company focus is on innovative design combined with precision engineering that enable development and manufacture of the next generation of casing and completion tools for the oil and gas industry. Deep Casing Tools Turbocaser uses a motor powered by drilling-mud to ream oil wells prior to pipe installation. Important to understand and optimise the performance of the multi-stage motor. Drilling mud is a non-Newtonian fluid with Reynold's Numbers between 4,000 and 20,000 and densities typically between 1.1 and 1.4 g/cm 3. Very computationally expensive to performa multi-stage design studies. Ideal application for HPC: project modelled mud flow through various Turbocaser turbine designs using HECToR and OpenFOAM eDIKT 2010

7. OpenFOAM-based design study Built mesh from existing Deep Casing Tools' AutoCAD designs Initial single stage simulations Multistage design study OpenFOAM Free, open source CFD software package Large user base across commercial and academic organisations. Solves complex fluid flows involving chemical reactions, turbulence and heat transfer, solid dynamics and electromagnetics. Already ported to HECToR, also runs on commodity resources eDIKT 2010

8. Project outcome Impact for company –Validation of basic design –Better understanding of product behaviour –Optimisation of pressure drop versus torque across multistage turbine –Support for future product design and development Impact for EPCC –Demonstration of value of HPC to real world problem –Increased staff experience of Computational Fluid Dynamics –Extended use of OpenFOAM on complex problem –Potential for further use of HECToR eg. rotating mesh eDIKT 2010

9. Prospect – Wind & Wave Modelling The Company: Prospect is an engineering design and analysis provider to the world energy industries. Headquartered in Aberdeen, Prospect was founded by in 1999 and has grown rapidly since to become part of a 300 strong Group with operations spanning six continents The Problem: Simulations exist of blades, turbine, wind on tower and waves on tower. Combining these simulations, using different software packages, is very difficult Massive computational resources required to keep all simulations sychronised Ideal application for HPC: EPCC asked to couple simulations and run result on HECToR © Copyright Davagh and licensed for reuse under this Creative Commons LicenceDavaghCreative Commons Licence eDIKT 2010

10. HPC solution: Coupled Simulation Create polyhedral / tetrahedral mesh to represent physical objects Simulate action of waves, ocean current and wind simultaneously Understand combined complex stresses on structure Tools: Simulia-Abaqus Finite Element Analysis StarCCM Computational Fluid Dynamics Tower Waves Current Wind eDIKT 2010

11. Prospect Summary Impact for Company First full simulation of entire deep- sea wind turbine In-house capacity to run 1 variation per night – HECToR can run 500 variations simultaneously Understanding impact of design changes, highlights efficiency savings Impact for EPCC Breakthroughs in coupled simulations – FEA and CFD Gained experience of two industry codes, ported to HPC platforms Potential to perform further work for client and software vendors © Copyright Davagh and licensed for reuse under this Creative Commons LicenceDavaghCreative Commons Licence eDIKT 2010

12. Challenges facing modelling & simulation We are at a complex juncture in the history of supercomputing For the past 20 years supercomputing has hitched a lift on the microprocessor revolution driven by the PC Hardware has been surprisingly stable EPCC in 1994 had the 512 processor Cray T3D system –0.0768 TFlops peak EPCC in 2010 retired the 2,560 processor IBM HPCx system –15.36 TFlops peak – 200 x faster but only 5 x more processors... But now we have a problem... eDIKT 2010

13. Average job size in Europe today eDIKT 2010 Data from PRACE Project

14. Multicore and Moores Law In 2005 microprocessor clock speeds peaked around 4GHz Exponential increase in individual core performance has ceased – in fact individual performance is reducing –Serial codes are now running slower on your new system Theoretical performance of multi-core microprocessors continues to rise Moores Law is alive and well –Relates to number of transistors on an area of silicon not clock speed But – todays microprocessors are more and more difficult to use –This can only get worse in coming years eDIKT 2010

15. Meeting user needs? Hardware is leaving many HPC users and codes behind Majority of codes scale to less than 512 cores –these will soon be desk-side systems Less than 10 codes in EU today will scale on capability systems with 100,000+ cores –Soon HECToR will have 70,000+ cores –Germanys Jugene system already has 294,912 cores Many industrial codes scale very poorly – some codes will soon find a laptop processor a challenge! Much hope is pinned on accelerator technology –But this has its own set of parallelism and programming challenges eDIKT 2010

16. The hardware software disconnect We have lived through a golden age of easy programmability and relaxed parallelism In the future codes will have to be highly scalable –But many were designed in a different age Main parallel programming paradigms were settled on 15-20 years ago –Many codes are written in Fortran, C or C++ –Most use either MPI or OpenMP –They use mathematical algorithms developed for 10s or 100s of processors – not hundreds of thousands Exascale systems now being planned will have many millions of cores eDIKT 2010

17. Reconnecting We need to focus much more effort on applications programming Rethinking the mathematics behind modelling on massively parallel systems Redesigning and re-implementing simulation and modelling codes Building new languages and methods of expressing and harnessing parallelism Unless we do this the justification for buying larger and more powerful systems will become increasingly false eDIKT 2010

18. Final comments The software area is huge We have challenges in The next few years are going to be very exciting in HPC eDIKT 2010 –Scaling –Languages –Tools –Power consumption –Accelerators –ISVs –Algorithms –Programming models –Memory management –Communications etc

19. Questions? eDIKT 2010