1. Performance Analysis, Profiling and Optimization of Weather Research and Forecasting (WRF) model
Negin Sobhani 1,2, Davide Del Vento2,David Gill2, Sam Elliot3,2,and Srinath Vadlamani4
1 University of Iowa
2National Center for atmospheric Research(NCAR)
3University of Colorado at Boulder
4Paratools Inc

2. Outline Introduction WRF MPI Scalability Hybrid Parallelization
Profiling WRF
Intel Vtune Amplifier XE
Taul tools
Identifying hotspots and suggested areas for improvement

3. The Weather Research & Forecasting(WRF) Model
Numerical weather prediction system
Designed for both operational forecasting and atmospheric research
Community model with large user base:
More than 30,000 users in 150 countries
Figure from WRF-ARW Technical Note

4. Previous Scaling Studies
WRF has benchmarked on different systems.
Figures from cisl.ucar.edu

5. TACC Stampede Supercomuter
Aggregate Peak Performance : ~10 PFLOPS(PF)
6400+ Dell PowerEdge (C8220z) server nodes
2 Intel Xeon E5 (Sandy Bridge) processors and an Intel Xeon Phi Coprocessor (MIC Architecture) per each compute Node
Each computer node has 32 GB of “host” memory with an additional 8GB of memory on the Xeon Phi coprocessor card
2.2 PF from Xeon E5 processors and 7.4 PF from Xeon Phi coprocessors
Figures from tacc.utexas.edu

6. Hurricane Sandy Benchmark
Coarser resolution 40-km (50x50)
Time-step: 180 sec
Finer resolution 4-km (500x 500)
Time-step: 20 sec
Time Period for both Simulation:
54 hour forecast
Between 2012 Oct 27 12:00 UTC through Oct 29 18:00 UTC
60 vertical layers

7. Scalability Assessment (MPI Only)
500X500 horizontal grids
MPI Bound
Compute Bound
Simulation Speed is duration of simulation per wall clock time

8. Scalability Assessment (MPI Only)
Allinea Perfomance Reports
Separate netcdf output file
(io_form_history=102 in WRF namelist)
79% of total time spent on MPI
I/O is calculated into MPI
I/O - allenia PR calculating
Wrf_opt=102
87% of total time spent on computation

9. Domain Decomposition (MPI only)
Per Grid :1/4 Computation and 1/2 MPI

10. AVX compiler flag
AVX (Intel® Advanced Vector Extensions) is a 256 bit instruction set extension
More aggressive optimization
Not working on intel 15
This issue has been reported to Intel
Intel 15 is a little bit better than this!

11. Hybrid Parallelization
Hybrid : distributed and shared memory parallelism(dmpar+smpar)
As the number of threads increase the performance decreases
The cores have never been oversubscribed
Binding increases the performance significantly
I_MPI_PROCESSOR_LIST= p1,p2
tacc_affinity script

12. Intel Vtune Amplifier XE
Intel profiling and performance analysis tool
Profiling includes stack sampling, thread profiling and hardware event sampling
Collect performance statistics of different part of the code

13. What does make WRF expensive?
Radiation
Longwave Radiation Scheme
RRTMG Scheme (ra_lw_physics =4)
Shortwave Radiation Scheme
CAM Scheme(ra_sw_physics = 3)
Microphysics Scheme
Thompson et al (mp_physics =8) .
But is this case representative of the significant effect of the dynamics on performance?
Time(%)

14. Microphysics options summary
Scheme
mp_physics
Simulation Speed
# of Variables
#timesteps/s
Kessler 1
2493.6 3
13.8
Purdue Lin et al. 2
2043.8 6
11.3
WSM-3
2263.8
12.5
WSM-5 4
2012.3 5
11.2
Ferrier(current NAM)
2451.2
13.6
WSM-6
1859.5
10.3
Goddard 6 class 7
1929.9
10.6
Thompson et al. 8
1739.8
9.7
Milbrandt- Yau 2-moment 9
1189.3
 13
6.6
Morison 2-moment 10
1475.5
8.2
WDM-5 14
1478.6  8
WDM-16 16
1358.8  9
7.5
Thompson microphysics is among the most expensive microphysics and it is widely used.

15. TAU tools
TAU (Tuning and Analysis Utilities) is a program and performance analysis tool framework for high-performance parallel and distributed computing
TAU can automatically instrument source code using a package called PDT for routines, loops, I/O, memory, phases, etc.
Tau uses wallclock time and PAPI metrics to read hardware counters for profiling and tracing

16. Using Tau/PAPI for Advection Module
1- PDT instrumentation for module_advect_em
2- Manually instrumented code for higher granularity of desired loops
TAU/PAPI variables analyzed:
Time
L1 and L2 Data Cache Misses (DCM)
Conditional branch instructions mispredicted
Floating point instruction and operations
Single and double precision vector/SIMD instructions

17. Identified Hotspots 1- Positive Definite Advection Loop (32 lines)
High Time
High cache misses (both L1 and L2 Cache misses)
High branch miss-prediction
2- x, y, z flux 5 advection equation loops
High Cache misses
Repeated through the code for different advection schemes

18. Moisture transport in ARW
Until recently, many weather models did not conserve moisture because of the numerical challenges in advection schemes.  high bias in precipitation
WRF-ARW scheme is conservative but not all of the advection schemes are.
This introduces new masses to the system.
Figure from Skamarock and Dudhia 2012
Advection schemes can introduce both positive and negative errors particularly at sharp gradients

19. Advection options in WRF
moist_adv_opt =0
moist_adv_opt =1
moist_adv_opt =2
Explicit IFs to remove oscillations
Explicit IFs to remove negative values and over shoots
High number of explicit IFs are causing high branch mispredictions
Figure from Skamarock and Dudhia 2012

20. The effect of optimization of advection module
1- Optimizing the positive definitive advection module
Test1 : WRF only case
Test 2: WRF-Chem case
Case
Advected Variables
Maximum performance increase
WRF
Moisture
13%
WRF-Chem
Moisture- Tracers- Species-Scalars- Chemical concentration- Particles
21% *
* The performance increase will be even significantly higher for dust and particle only WRF-Chem cases.
This hotspot has a potential for being optimized and provides significant improvement in performance.

21. Identified Hotspots 1- Positive Definite Advection Loop
High Time
High cache misses (both L1 and L2 Cache misses)
High branch miss-prediction
2- x, y, z flux 5 advection equation loops
High Cache misses
Repeated through the code for different advection schemes

22. The effect of optimization of advection equations
2- Flux 5 advection equations
High Time and High L1 and L2 Data Cache Misses
This loop is repeated throughout the code for x, y and z directions
Very similar loop repeated for all the advection schemes
Test1 : WRF 4 km benchmark with TAU instrumentation
58% time spent in advection is in these flux equations loops
Many L1 data cache misses per iteration
Many L2 data cache misses per iteration
This hotspot has a potential for being optimized and provides significant improvement in performance.

23. Conclusion WRF shows good MPI scalability depending on the workload
Thread Binding should be used for improving the performance of the WRF hybrid runs
Intel Vtune Amplifier and Tau tools used for performance analysis of WRF code.
Dynamics is identified as the most expensive part of ARW
We identified the hotspots of the advection module and estimated the amount of performance increase from modifying these parts of the WRF code

24. Ongoing and Future Work
Performance Improvement of advection module
Analysis of hardware counters to fix branch mispredictions and cache misses
Advection module vectorization for Intel Xeon Phi Coprocessors
Reducing memory footprint by decreasing the number of temporary variables
Exploring performance optimization with different compiler flags
Loop transformation for enabling better vectorization

25. Acknowledgements Davide Del Vento Rich Loft Srinath Vadlamani
Dave Gill
Greg Carmichael
All SIParCS admins and staff
why

26. Microphysics Schemes Provides atmospheric heat and moisture tendencies
Includes water vapor, cloud and precipitation processes
Microphysical rates
Surface rainfall
Mielikainen et al. 2014

27. WRF Model Integration Procedure
Begin time step
Runge-Kutta loop (steps 1, 2, and 3)
(i) advection, p-grad, buoyancy using
(ii) physics if step 1, save for steps 2 and 3
(iii) mixing, other non-RK dynamics, save…
(iv) assemble dynamics tendencies
Acoustic step loop
(i) advance U,V, then W,
(ii) time-average U,V,W
End acoustic loop
Advance scalars using time-averaged U,V,W
End Runge-Kutta loop
Other physics (currently microphysics)
End time step