1. NC STATE UNIVERSITY Prediction of Optical Scale Turbulence with a Typical NWP Code: Lessons Learned (?) Projected to Petascale Computing D. Scott McRae Aerospace Engineering North Carolina State University NCAR Theme of the Year Workshop May 6, 2008

2. NC STATE UNIVERSITY Acknowledgments Prior funding: –The US Army Research Laboratory, Battlefield Environments Division, WSMR and the HELJTO monitored by Dr. David Tofsted, ARLWSMR Current funding: –The US Air Force Research Laboratory, Space Vehicles Directorate, Hanscom AFB, MA through contract FA8718-04-C-0019; monitored by Dr. George Jumper, AFRL/VSBYA –NorthWest Research Associates, CORA division, monitored by Dr. Joe Werne Contributors – Xudong Xiao, H. A. Hassan, and Yih-Pin Liew, NCSU; Talat Odman, GIT; and Frank Ruggiero,AFRL

3. NC STATE UNIVERSITY Outline Current goal –Prediction of Optical and Clear Air Turbulence by modifying existing NWP codes to increase prediction accuracy Approach Examples of current work Projection to Petascale Analysis of solution Numerical Issues Concluding Remarks

4. NC STATE UNIVERSITY What is optical turbulence? The term “Atmospheric Optical Turbulence” refers to fluctuations in the refractive index of air due to turbulence in the atmosphere. –Affects optical propagation by random refraction –Reduces the effective power of optical signals Quantitative measure of the intensity of atmospheric optical turbulence: structure parameter of refractive index, C n 2, Integration of an accurate prediction of C n 2 along the beam/viewing path is the primary need for many optical systems Accurate prediction requires well resolved dynamics and physics, whether from observation or from atmospheric models. In the latter case, physically accurate turbulence models are required for the scales unresolved by the atmospheric model

5. NC STATE UNIVERSITY Observation versus Clear 1 Model Jumper, Beland, 2000

6. NC STATE UNIVERSITY Approach Prediction usually required for sub-mesoscale domains –Radiosonde soundings –Numerical weather prediction codes with parameterizations for –DNS simulation –Statistical techniques using many sources Present approach- modify existing NWP codes to increase accuracy of prediction –LES scale Prediction Using Dynamic 3-D Adaptive Grid –Hybrid LES/RANS Turbulence Model with direct Output- described by Hassan in a later talk Resolution of shear requires adaptation in all three coordinate directions Accuracy of prediction can be increased by including more physics of turbulence in the model

7. NC STATE UNIVERSITY Model Modifications NCSU r-refinement Dynamic Solution Adaptive Grid Algorithm (DSAGA) –Resolve selected features/characteristics/properties dynamically –Criteria selected initially for resolution –Code determines location and resolution automatically –Adapts in all three dimensions The NCSU k- hybrid turbulence model –Four equations based on exact equations derived from the Navier-Stokes and modeled term by term- described by Hassan in a following talk

8. NC STATE UNIVERSITY Results – 2D case Geometry of the computational domain Mesh size: 221 X 126 Same setup as in Ref. 14 (by Doyle etc.)

9. NC STATE UNIVERSITY Results – 2D case Inflow wind speed profile from the Grand Junction, CO, sounding for 1200 UTC 11 January 1972

10. NC STATE UNIVERSITY Results – 2D case(MILES) Adaptive mesh at t=3h

11. NC STATE UNIVERSITY Results – 2D(MILES) Potential temperature contours

12. NC STATE UNIVERSITY Numerical Lidar (x= -2km)

13. NC STATE UNIVERSITY Velocity vectors- 3hours

14. NC STATE UNIVERSITY Detail- Velocity vectors- 3 hours

15. NC STATE UNIVERSITY “Isolated “ vortices

16. NC STATE UNIVERSITY Velocity vector animation

17. NC STATE UNIVERSITY Cn 2 contours and balloon trajectory

18. NC STATE UNIVERSITY Comparison of zonal wind profile

19. NC STATE UNIVERSITY Comparison of zonal wind profile

20. NC STATE UNIVERSITY Comparison of meridional wind profile

21. NC STATE UNIVERSITY Comparison of meridional wind profile

22. NC STATE UNIVERSITY Comparison of

23. NC STATE UNIVERSITY Comparison of vertical spacing

24. NC STATE UNIVERSITY Weight Function Along Trajectory

25. Snapshot of the Adaptive Grid: Tennessee Valley Ozone Simulation Grid adapting to density of NO x plumes From Dabberdt W. F. et al., “Meteorological Research Needs for Improved Air Quality Forecasting” Bulletin of the American Meteorological Society, vol. 85, no.4, pp. 563-586, April 2004.

26. Superior O 3 Predictions with Adaptive Grid Sumner Co., TN Graves Co., KY

27. NC STATE UNIVERSITY Scaling Adaptive MM5 to Franklin

28. NC STATE UNIVERSITY Adaptation plus Petascale Adaptive meshing will move specific resolutions upwards on the previous chart Dynamic adaptation may provide a more efficient alternative to standard nesting –Targeted solution dependent weight function determines automatically where resolution is needed rather than a predetermined nest structure –Initial budget mapped uniformly unto processors –No mesh boundary errors (however, adaptation is not without error) Unfortunately, present NWP codes are unlikely to scale to full use of a petascale resource due to a combination of factors

29. NC STATE UNIVERSITY Evaluation of MM5 Results Filtering/dissipation in code tends to reduce or eliminate structure/frequencies needed for prediction Solution does not converge as mesh is refined- S. Koch, NOAA Approximately 3-1 reduction in vertical spacing improves structure but still appears over damped. Benefit due to local mesh refinement difficult to assess –LES resolution of turbulence scales not yet achieved (Terra Incognita – Wyngaard) WRF-ARW shares basic MM5 approach with updated algorithms- has 8 identifiable dissipation sources

30. NC STATE UNIVERSITY Numerics- MM5 Horizontal integration scheme- time centered explicit (leapfrog) –Neutrally stable for all, for central space –MM5 filtering (Asselin, 1972) for any variable α: Where for all conditions –Staggered grid with averaging

31. NC STATE UNIVERSITY Numerics –Vertical coordinate –Where –The turbulence model provides an alternative Eddy viscosity

32. NC STATE UNIVERSITY Numerics Vertical integration scheme- semi-implicit (Klemp and Wilhelmson, 1978) –horizontal results held constant, With divergence dampingadded

33. NC STATE UNIVERSITY Assessment of Damping prediction derived from local state and spatial variation- spatial filtering is then the issue Asselin filter used to stabilize Leapfrog in MM5 Where for all conditions This can be expressed as Assuming linear advection

34. NC STATE UNIVERSITY Assessment of Damping Which becomes This implies that spatial damping due to the Asselin filter remains constant relative to the mesh if and CFL remain constant as the mesh spacing is reduced This filter contributes to mathematical non-convergence

35. NC STATE UNIVERSITY Assessment of Damping Divergence Damping (Skamarock and Klemp, 1992) –Essentially a modification of the normal stress term in the diagonal of the stress tensor (“Normal Stress Damping”, McRae, 1976) –Simplified analysis as in Durran – x momentum where Discretizing the first term gives

36. NC STATE UNIVERSITY Assessment of Damping This is a spatial damping that remains constant relative to the mesh as the mesh spacing is reduced This damping contributes to mathematical non- convergence Furthermore, divergence damping- –Inserts a modified normal stress into the Euler equations with an unscaled coefficient of the order of an eddy viscosity –Interacts with turbulence models/parameterizations resulting in non-physical shear layers –Biases the pseudo- incompressibility formulation

37. NC STATE UNIVERSITY Evaluation of MM5 Results Assessing sensitivity of convergence to individual dissipations time consuming and problematical Needed- A technique for assessing the net effect of all of the dissipations applied in the course of the integration –A posteriori ( forensic ) analysis of the solution The mesh is a band pass filter and determines the number of Fourier terms available for constructing the solution

38. NC STATE UNIVERSITY Total Assessment of Dampinfg Use Fourier analysis of the solution to ascertain, approximately, how much of the spectrum theoretically resolved by the mesh remains in the solution (ignoring aliasing) –Sample the solution in a direction in which resolution is important –Use discrete FFT to obtain approximate frequency versus amplitude distribution –Infer overall filtration

39. NC STATE UNIVERSITY Sampling

40. Sampled Function

41. NC STATE UNIVERSITY FFT Output

42. NC STATE UNIVERSITY Three coordinate dynamic grid adaptation in conjunction with physically based turbulence modeling results in demonstrated improvement in optical turbulence prediction However, damping and filtering in present NWP codes limit the improvement Some damping types, as used, contribute directly to mathematical non-convergence and may lead to non- physical shear layers, thereby damaging further the optical turbulence prediction Concluding Remarks

43. NC STATE UNIVERSITY Achieving full advantage from use of Petascale computing resources will likely require major changes in the current NWP code equations, algorithms and their implementation Concluding Remarks

44. NC STATE UNIVERSITY We are grateful for the many helpful conversations with the people of AFRL/VSBYA, ARL/WSMR, NWRA, DRI, NCSU MEAS, NCAR and others Concluding Remarks