1. Development and Application of Parallel Plume-in-Grid Models Prakash Karamchandani, Krish Vijayaraghavan, Shu-Yun Chen and Christian Seigneur AER, San Ramon, CA 7th Annual CMAS Conference October 6–8, 2008 Chapel Hill, NC

2. Limitations of Traditional Grid Modeling Horizontal resolution of a few kilometers to tens of kilometers cannot resolve sub-grid scale effects such as transport and chemistry of point source emissions Artificial dilution of stack emissions Unrealistic near-stack plume concentrations Incorrect representation of plume chemistry Incorrect representation of plume transport Plume Size vs Grid Size (from Godowitch, 2004)

3. Plume Chemistry & Relevance to Ozone and PM Modeling Early Plume Dispersion NO/NO 2 /O 3 chemistry 1 2 Mid-range Plume Dispersion Reduced VOC/NO x /O 3 chemistry — acid formation from OH and NO 3 /N 2 O 5 chemistry Long-range Plume Dispersion 3 Full VOC/NO x /O 3 chemistry — acid and O 3 formation

4. Plume-in-Grid (PiG) Modeling Plume-in-Grid (PiG) approach provides a sub-grid scale representation of stack plumes Addresses inability of 3-D grid models to correctly simulate atmospheric fate of stack emissions Approach consists of embedding a reactive puff model within the grid model The initial transport and chemistry of point source emissions are treated with the puff model When puff sizes are comparable to grid model resolution, the puffs are merged with the grid and subsequent calculations are done with the grid model

5. AMSTERDAM Advanced Modeling System for Transport, Emissions, Reactions & Deposition of Atmospheric Matter Suite of models based on CMAQ v 4.6, October 2006 release and with alternate options Options: –MADRID treatment for PM: Model of Aerosol Dynamics, Reaction, Ionization and Dissolution –APT: Advanced Plume Treatment with embedded plume model SCICHEM (state-of-the science treatment of stack plumes at the sub-grid scale) APT can be used with either the MADRID treatment for PM or the CMAQ treatment (currently available for AERO3 option)

6. AMSTERDAM Components CMAQ v. 4.6 MADRID PM Treatment CMAQ-MADRID SCICHEM-AERO3 PM Treatment based on EPA CMAQ SCICHEM-MADRID PM Treatment based on CMAQ-MADRID CMAQ-MADRID-APT CMAQ-AERO3-APT

7. SCICHEM Three-dimensional puff-based model Second-order closure approach for plume dispersion Puff splitting and merging Treatment of plume overlaps Optional treatment of building downwash Optional treatment of turbulent chemistry PM, gas-phase and aqueous-phase chemistry treatments consistent with host model

8. Model Applications/Evaluations North-eastern U.S. with two nested grid (12 km & 4 km) domains (NARSTO) –5 day episode (O 3 only) –30 point sources simulated explicitly –PiG model about 1.5 times slower than grid model Central California (4 km resolution) –3 day episode (O 3 only) –10 point sources simulated explicitly –PiG model about 1.2 times slower than grid model

9. Model Applications/Evaluations (continued) Eastern U.S. (VISTAS 12 km domain) –2 month simulations for O 3 and PM –14 point sources simulated explicitly –PiG model about 1.25 times slower than grid model Southeastern U.S. (ALGA 12 km domain) –Annual simulations –O 3, PM, Hg and nitrogen deposition –40 point sources simulated explicitly –PiG model about 1.8 times slower than grid model

10. Typical Results The PiG treatment has a strong effect on model predictions of surface O 3 titration (near large NO x point sources), as well as O 3, sulfate and nitrate formation downwind of large NO x and SO 2 point sources A purely gridded approach typically overestimates PM production downwind of large NO x point sources because it overestimates SO 2 to sulfate and NO x to nitrate conversion rates near the stack Overall model performance statistics are almost identical between the grid-only and PiG treatments, but observed plume events are better captured in the PiG approach than in the purely gridded approach

11. Power-Plant Contributions to 24-hr Average Sulfate Concentrations CMAQ-MADRIDCMAQ-MADRID-APT

12. Change in Power-Plant Contributions (%) to PM 2.5 Sulfate Concentrations When a PiG Approach is Used

13. Conversion of Power Plant SO 2 Emissions Domain-wide mass-budget analysis performed for SO 2 and sulfate attributable to power plant emissions Sulfate to Total Sulfur Ratios (%) Emissions CMAQ-MADRIDCMAQ-MADRID-APT January 2.2517.815.6 July 2.3575.567.4 Approximate SO 2 Conversion (%) CMAQ-MADRIDCMAQ-MADRID-APTChange January 15.913.7-14% July 74.966.6-11%

14. Spatial Distribution of Total Nitrogen Deposition Change in annual dry + wet deposition flux due to power plant NO x controls CMAQ-MADRID CMAQ-MADRID-APT APT: Less oxidation of NO x to HNO 3 => Less dry deposition near the plant Maximum reduction in deposition flux 0.85 kg/ha 0.42 kg/ha

15. PiG Modeling Constraints Can be computationally expensive if a large number of point sources are treated with the puff model – computational requirements increase by a factor of two to three for 50 to 100 sources Point sources have to be selected carefully to limit the number of sources treated To obtain results in a reasonable amount of time, annual simulations are usually conducted by dividing the calendar year into quarters and simulating each quarter on different processors or machines Parallel version of code can address these constraints

16. Parallelization of APT Parallelization of stand-alone version of puff model (SCICHEM) Adaptation of parallel stand-alone SCICHEM to parallel plume-in-grid version Development of appropriate parallel interfaces between parallel CMAQ-MADRID/CMAQ-AERO3 and parallel plume-in-grid version of SCICHEM

17. Parallelization of Host Models CMAQ and CMAQ- MADRID –Based on Message Passing Interface (MPI) –Horizontal domain decomposition

18. Parallelization of SCICHEM SCICHEM –Puff decomposition for chemistry step –Needs access to entire 3-D grid –Uses MPI

19. Parallel Interface Gathers 3-D sub-domain concentrations from the various processors to create a global 3-D concentration array Provides global 3-D concentration array to SCICHEM After the SCICHEM time step, the modified global 3-D concentration array is scattered to the 3-D sub-domain concentration arrays

20. Model Interaction Diagram Domain, grid information Geophysical data Meteorological data Deposition velocities Parallel CMAQ-MADRID/ CMAQ-AERO3 Parallel SCICHEM Emissions, IC/BC Output concentrations, Deposition, Puff diagnostics Output puff information Point source emissions Merge puffs I/O API I/O API I/O API I/O API Standard SCICHEM output Puff diagnostics Control File Parallel Interface

21. Current Status Development of parallel version of AMSTERDAM completed in 2008 On a 4-processor machine, the parallel version is about 2.5 times faster than the single-processor version On-going project to apply the model to the central and eastern United States at 12 km resolution and to evaluate it with available data –Over 150 point sources explicitly treated with APT –Annual actual and typical simulations for 2002 –Future year emission scenarios –Other emission sensitivity scenarios

22. Ongoing Application of Parallel PiG Model 12 km grid resolution 243 x 246 x 19 grid cells Over 150 PiG sources

23. Acknowledgements Funding for model development, including parallelization, and model application: –EPRI Collaboration on parallelization: –L-3 Communications Titan Group Parallelization insights: –Dr. David Wong, EPA