HPC need and potential of ANSYS CFD and mechanical products at CERN A. Rakai EN-CV-PJ2 5/4/2016

Outline «New» CFD project ANSYS HPC strategy Relevant HPC at CERN 5/4/2016 A. Rakai EN-CV-PJ3

Outline «New» CFD projects ANSYS HPC strategy Relevant HPC at CERN A. Rakai EN-CV-PJ4 5/4/2016

«New» CFD project 1: accidental release modelling Gaussian plume type: uniform flow field Computational fluid dynamics: resolving flow field A. Rakai EN-CV-PJ5 Gaussian plume Computational fluid dynamics From COST Action ES1006 5/4/2016

Demonstrative calculations on a smaller domain: Only a few buildings around Isolde: 6A. Rakai EN-CV-PJ Pollutant source next to a building Far field wind direction 5/4/2016

Devil is in the details: close to source – iso-surface view 7A. Rakai EN-CV-PJ Wind 5/4/2016

«New» CFD project 2: ATLAS cavern ventilation New technical student: Eugenia Rocco A. Rakai EN-CV-PJ8 5/4/2016

CERN HPC needs in general Initiative from G. Arduini (BE-ABP) to collect similar needs and present to F. Hemmer: P. Collier – BE R. Jones – BE-BI E. Jensen – BE-RF S.Gilardini – EN-STI M. Battistin – EN-CV F. Bertinelli, F. Carra – EN-MME P. Chiggiato – TE-VSC L. Bottura, Duarte Ramos – TE-MSC A. Rakai EN-CV-PJ9 5/4/2016

Outline «New» CFD projects ANSYS HPC strategy Relevant HPC at CERN A. Rakai EN-CV-PJ10 5/4/2016

ANSYS HPC strategy – scaling A. Rakai EN-CV-PJ11 5/4/2016 Scales well up to ~ cores Good scaling with 5500 cells/core: Interest for even small CFD (1million cells) models to run on ~200 cores

ANSYS HPC strategy – scaling A. Rakai EN-CV-PJ12 Good scaling with 5500 cells/core: Interest for small (1million cells) models Also for mechanical applications until ~100 cores 5/4/2016

ANSYS HPC strategy – interconnection A. Rakai EN-CV-PJ13 5/4/2016

ANSYS HPC strategy – cloud A. Rakai EN-CV-PJ14 5/4/2016

ANSYS HPC strategy – benchmarking A. Rakai EN-CV-PJ15 5/4/2016 Automatic toolkit with Perl script to be run from installation directory, ~60 GB disc space is required

Outline «New» CFD projects ANSYS HPC strategy Relevant HPC at CERN A. Rakai EN-CV-PJ16 5/4/2016

History of IT CFD meetings A. Rakai EN-CV-PJ17 5/4/2016 Single machine, several CPUs No hyperthreading Low latency interconnection Linux better than windows

Benchmarking 2013: 18A. Rakai EN-CV-PJ 5/4/2016

CERN HPC – EngPara linux Low latency ethernet cards (Infiniband?) Batch runs with MPI 12 cores/node, 64Gb of memory Lxbatch/eng, only for parallel applications For two user groups: CFD and BE users Long experience within CFD group (logging to nodes, check availability) but people have left A. Rakai EN-CV-PJ19 5/4/2016

CERN HPC – HpcEng Win cEngCluster Windows Server 2012 R2 and Windows HPC Pack Total 1024 cores Low latency interconnect (Infiniband?) Only 1 year of experience (no memory use or availability check), but for new users more accessible A. Rakai EN-CV-PJ20 5/4/2016

CERN HPC – HpcEng Win cEngCluster Windows Server 2012 R2 and Windows HPC Pack 16x32 core, 512 GB RAM 32x16 core, 128 GB RAM Only 1 year of experience, but for new users more accessible A. Rakai EN-CV-PJ21 How to know which one is free, the ones taken use how much memory? 5/4/2016

Benchmarking 2015: 22A. Rakai EN-CV-PJ Speed-up comparison of LSF and Windows cluster in 2015: HVAC case, R15.0 Ideal speed-up: linear, a calculation runs n times faster on n cores 5/4/2016

Benchmarking 2016 – winHPC: 4/21/16 23A. Rakai EN-CV-PJ HVAC case, R15.0 HVAC case, R16.2 Outdoor case, R16.2 Not able to repeat last year’s performance

Benchmarking 2016 – PC: 24A. Rakai EN-CV-PJ HP Z800 workstation: 24GB RAM, 8 core Intel Xeon CPU 5/4/2016 Far from ideal performance

CERN ANSYS license monitoring A. Rakai EN-CV-PJ25 Max cores to be used theoretically for one job with existing licenses: 16 (aa_r_cfd) (aa_r_hpc) = 400 HPC is not so widely used apparently (could not find stat) 5/4/2016

A. Rakai EN-CV-PJ26 5/4/2016 The only license usage statistics I found from 2015: EN-CV with 1 person using ANSYS uses a lot

Conclusion 1 CFD and other (BE department) codes have more (theoretical – thousands of cores) potential than what CERN HPC capacity can provide at the moment, and it is growing Windows or linux based cluster: both have advantages and disadvantages Investigating cloud possibilities can be interesting for optimizing usage A. Rakai EN-CV-PJ27 5/4/2016

Conclusion 2 CFD calculations can benefit from this growing parallel capacity: Faster results More (numerically) accurate results More scenarios investigated A. Rakai EN-CV-PJ28 5/4/2016

ANSYS HPC strategy – best practices A. Rakai EN-CV-PJ30 1. Don’t Move the Data (More Than You Have to) 2. Remote Graphics & Graphics User Interface for End-to-End Simulation 3. Secure Network Communications & Data Storage 4. Effective End-User Access for Job & Data Management 5. Re-Use On-Premise Licenses (or Not) 6. Consider a Mix of Business Models 7. Match Your Cloud to Your HPC Workload 8. Start Small, Grow Organically… but Think Big 5/4/2016

ANSYS HPC strategy – tests A. Rakai EN-CV-PJ31 Technical Computing Power When You Need It UberCloud is the online community and marketplace where engineers and scientists discover, try and buy Computing as a Service UberCloud Experiment: a free experiment of 1000 free CPU hours with professional providers (ANSYS) and experts, possibility to bring licenses 5/4/2016