Samm Elliott Mentor – Davide Del Vento WRF Performance Optimization Targeting Intel Multicore and Manycore Architectures Samm Elliott Mentor – Davide Del Vento

The WRF Model The Weather Research and Forecasting (WRF) Model is a mesoscale numerical weather prediction system designed for both atmospheric research and operational forecasting needs. Used by over 30,000 Scientists around the world. Any Optimizations that can be made will make a significant impact on the WRF community as a whole.

Stampede Supercomputer 6400 Nodes (Dell PowerEdge C8220) Two CPU’s Per Node (Intel Xeon E5-2680 Processors Per Node) 32 GB Memory Per Node 1-2 Coprocessors Per Node (Intel Xeon Phi SE10P)

Intel Xeon vs Xeon Phi Architecture Xeon E5 2680 CPU 8 Cores 2-Way Hyperthreading 256 Bit Vector Registers 2.7 GHz 32 KB L1 256 KB L2 20 MB Shared L3 32 GB Main Memory Xeon Phi SE10P Coprocessor 61 Cores 4-Way Hyperthreading (2 Hardware Threads Per Core) 512 Bit Vector Registers 1.1 GHz 512 KB L2 No L3 Cache 8 GB Main Memory

Intro: WRF Gets Good Performance on Xeon Phi! Memory Limitations

Standard MPI Implementation Strong Scaling Low Efficiency Fast Time to Solution High Efficiency Slow Time to Solution Compute Bound MPI Bound

MPI Tile Decomposition

Hybrid Tile Decomposition WRF does NOT use loop-level OpenMP parallelization MPI Tiles OpenMP Tiles

MPI vs. Hybrid Parallelization What do we expect? Less Halo Layer Communication (White Arrows) Less MPI Overhead and Therefore Faster! No loop level omp par MPI Only Hybrid

MPI vs. Hybrid Strong Scaling Hybrid parallelization of WRF is consistently better than strict MPI and is significantly better in the MPI bound regions.

Core Binding Using processor domain binding allows all OpenMP threads within an MPI task to share L3 cache. Socket 0 Socket 1 Rank 0 Rank 1 Socket 0 Rank 0 Remake bigger text Rank 1 Socket 1

pNetcdf Serial pNetcdf Separate Output Files Using separate output files (io_form_history=102) makes output write times negligible – very worthwhile!

Host Node Optimization Summary Hybrid Parallelization of WRF consistently gives better performance results than strict MPI Much faster than strict MPI in MPI bound region Using separate output files requires post-processing but kills any time spent in writing history Process/Thread binding is critical for hybrid WRF Take into consideration memory limitations for hybrid WRF and set environment variable OMP_STACKSIZE to avoid memory issues. Change pnetcdf

Xeon Phi IO Serial pNetcdf Separate Output Files

How Does Varying The Number of MPI Tasks Per Xeon Phi Effect The Performance? Total Number of Cores = 240 Less MPI Overhead More MPI Overhead Why are we seeing better performance with more MPI overhead? Specify OMP MPI core utilixation These results suggest that there are issues with WRF’s OpenMP strategies

Hybrid Tile Decomposition WRF does NOT use loop-level OpenMP parallelization MPI Tiles OpenMP Tiles

Open MP Imbalancing Two types of Imbalancing: Default Tiling Issue: 1. Number of OpenMP Tiles > Number of OpenMP Threads 2. OpenMP Tiles are Different Sizes Default Tiling Issue: When number of threads is equal to any multiple of the number of MPI tile rows

OpenMP Imbalance Type 1: Example – 10 x 10 grid run with 2 MPI tasks and 8 OpenMP threads per MPI task MPI Rank 0 MPI Rank 1 Threads #1 and #2 compute 2 OpenMP tiles each (twice as much work as other threads + context switch overhead)

OpenMP Imbalance Type 2: Example – 4 MPI Tasks, 4 OpenMP Threads Per Task MPI Rank 0 MPI Rank 1 MPI Rank 3 MPI Rank 2 Thread #4 computes twice as many grid cells as all other threads!

OpenMP Good Balancing Example: 8 by 8 grid run with 2 MPI Tasks and 8 OpenMP threads per task MPI Rank 0 MPI Rank 1 I was able to resolve this issue and will be fixed in the next version of WRF The logical tiling would be a 2 by 4 OpenMP tiling but WRF does a 3 by 4 creating an unnecessary imbalance

WRF OpenMP Tile Strategy

What is an Optimal WRF Case For Xeon Phi? Xeon Phi – Initial Scaling Xeon Phi Approaches Xeon Performance for Large Workload/Core a

Scaling Gridsize Xeon Phi Hits Memory Limits Consistent Balancing for Symmetric Runs Xeon Hits Performance Limit Far Before Xeon Phi Xeon Phi exceeds Xeon Performance for > 30,000 Horizontal Gridpoints/CPU-Coprocessor

Xeon Phi Optimization Summary: The Good Xeon Phi can be more efficient than host CPU’s in extreme high efficiency/slow time to solution region For highly efficient workloads, due to low MPI overhead and constant efficiency it is possible to have well balanced symmetric CPU-Coprocessor WRF runs that are significantly more efficient than running on either homogeneous architecture

Xeon Phi Optimization Summary: The Bad WRF strong scaling performance is significantly less than host node CPU runs. WRF tiling strategies are not well optimized for manycore architectures WRF/Fortran’s array allocation strategies result in much larger memory requirements and limit workloads that otherwise would have high efficiency on Xeon Phi

Xeon Phi Optimization Summary: The Ugly Although Xeon Phi could be used for highly efficient WRF simulations, finding the correct: problem sizes task-thread ratios tile decompositions and workload per core while taking into consideration memory limitations makes Xeon Phi extremely impractical for the vast majority of WRF users.

Why is it Important to Continue Researching WRF on Xeon Phi? Core counts will continue to increase for future HPC architectures – This will require better hybrid strategies for our models. Xeon Phi is very representative of these architectures and is a tool for exposing various issues that otherwise may not be noticed until further down the road.

Future Work Create better MPI+OpenMP tile strategies for low workload per core simulations Better understand performance issues with heap allocation and overall memory access patterns Assess any other concerns that may hinder WRF performance on future multicore/manycore architectures

Acknowledgements Davide Del Vento – Mentor Dave Gill – WRF Help Srinath Vadlamani – Xeon Phi/Profiling Mimi Hughes – WRF XSEDE/Stampede – Project Supercomputer Usage SIParCS/Rich Loft – Internship Opportunity Thank you all for giving me so much help throughout this process. I am extremely thankful for my experience here at NCAR!

Questions?

Memory Issues For Small Core Counts Memory use by WRF variables is multiplied by the number of threads per MPI task. Not much of an issue for small number of threads per task Ensure OMP_STACKSIZE is set correctly (default may be set much lower than necessary)

Memory Issues For Small Core Counts Memory use by WRF variables is multiplied by the number of threads per MPI task. Temporary Solution: Force heap array allocation (-heap-array) Extremely slow in compute-bound regions (“speeddown” proportional to number of threads per task) Potential Culprits: Cache Coherency? Repeated temporary array allocation?

Forcing Heap Array Allocation