1. Center for Space Environment Modeling http://csem.engin.umich.edu Gábor Tóth Center for Space Environment Modeling University of Michigan Gábor Tóth Center for Space Environment Modeling University of Michigan The Grand Challenge of Space Weather Prediction

2. Center for Space Environment Modeling http://csem.engin.umich.edu Tamas Gombosi, Kenneth Powell Ward Manchester, Ilia Roussev Darren De Zeeuw, Igor Sokolov Aaron Ridley, Kenneth Hansen Richard Wolf, Stanislav Sazykin (Rice University) József Kóta (Univ. of Arizona) Tamas Gombosi, Kenneth Powell Ward Manchester, Ilia Roussev Darren De Zeeuw, Igor Sokolov Aaron Ridley, Kenneth Hansen Richard Wolf, Stanislav Sazykin (Rice University) József Kóta (Univ. of Arizona) Collaborators Grants DoD MURI and NASA CT Projects

3. Center for Space Environment Modeling http://csem.engin.umich.edu What is Space Weather and Why to Predict It? Parallel MHD Code: BATSRUS Space Weather Modeling Framework (SWMF) Some Results Concluding Remarks What is Space Weather and Why to Predict It? Parallel MHD Code: BATSRUS Space Weather Modeling Framework (SWMF) Some Results Concluding Remarks Outline of Talk

4. Center for Space Environment Modeling http://csem.engin.umich.edu What Space Weather Means Conditions on the Sun and in the solar wind, magnetosphere, ionosphere, and thermosphere that can influence the performance and reliability of space-born and ground-based technological systems and can endanger human life or health. Space physics that affects us.

5. Center for Space Environment Modeling http://csem.engin.umich.edu Solar Activity...

6. Center for Space Environment Modeling http://csem.engin.umich.edu Affects Earth: The Aurorae

7. Center for Space Environment Modeling http://csem.engin.umich.edu Other Effects of Space Weather

8. Center for Space Environment Modeling http://csem.engin.umich.edu MHD Code: BATSRUS Block Adaptive Tree Solar-wind Roe Upwind Scheme Conservative finite-volume discretization Shock-capturing Total Variation Diminishing schemes Parallel block-adaptive grid (Cartesian and generalized) Explicit and implicit time stepping Classical and semi-relativistic MHD equations Multi-species chemistry Splitting the magnetic field into B 0 + B 1 Various methods to control the divergence of B Block Adaptive Tree Solar-wind Roe Upwind Scheme Conservative finite-volume discretization Shock-capturing Total Variation Diminishing schemes Parallel block-adaptive grid (Cartesian and generalized) Explicit and implicit time stepping Classical and semi-relativistic MHD equations Multi-species chemistry Splitting the magnetic field into B 0 + B 1 Various methods to control the divergence of B

9. Center for Space Environment Modeling http://csem.engin.umich.edu MHD Equations in Conservative vs. Non-Conservative Form Conservative form is required for correct jump conditions across shock waves. Energy conservation provides proper amount of Joule heating for reconnection even in ideal MHD. Non-conservative pressure equation is preferred for maintaining positivity. Hybrid scheme: use pressure equation where possible. Conservative form is required for correct jump conditions across shock waves. Energy conservation provides proper amount of Joule heating for reconnection even in ideal MHD. Non-conservative pressure equation is preferred for maintaining positivity. Hybrid scheme: use pressure equation where possible.

10. Center for Space Environment Modeling http://csem.engin.umich.edu Conservative Finite-Volume Method

11. Center for Space Environment Modeling http://csem.engin.umich.edu Limited Reconstruction: TVD Finite volume data stored at cell centers, fluxes computed at interfaces between cells. Need an interpolation scheme to give accurate values at the two sides of the interface. Reconstruction process can introduce new extrema. Need to limit the slopes so that reconstructed values are bounded by cell- center values. Limited reconstruction results in a total variation diminishing scheme First order near discontinuities and second order in smooth regions.

12. Center for Space Environment Modeling http://csem.engin.umich.edu Splitting the Magnetic Field The magnetic field has huge gradients near the Sun and Earth: – Large truncation errors. – Pressure calculated from total energy can become negative. – Difficult to maintain boundary conditions. Solution: split the magnetic field as B = B 0 + B 1 where B 0 is a divergence and curl free analytic function. – Gradients in B 1 are small. – Total energy contains B 1 only. – Boundary condition for B 1 is simple. The magnetic field has huge gradients near the Sun and Earth: – Large truncation errors. – Pressure calculated from total energy can become negative. – Difficult to maintain boundary conditions. Solution: split the magnetic field as B = B 0 + B 1 where B 0 is a divergence and curl free analytic function. – Gradients in B 1 are small. – Total energy contains B 1 only. – Boundary condition for B 1 is simple.

13. Center for Space Environment Modeling http://csem.engin.umich.edu Vastly Disparate Scales Spatial: Resolution needed at Earth: 1/4 R E Resolution needed at Sun: 1/32 R S Sun-Earth distance: 1AU 1 AU = 215 R S = 23,456 R E Temporal: CME needs 3 days to arrive at Earth. Time step is limited to a fraction of a second in some regions. Spatial: Resolution needed at Earth: 1/4 R E Resolution needed at Sun: 1/32 R S Sun-Earth distance: 1AU 1 AU = 215 R S = 23,456 R E Temporal: CME needs 3 days to arrive at Earth. Time step is limited to a fraction of a second in some regions.

14. Center for Space Environment Modeling http://csem.engin.umich.edu Adaptive Block Structure Each block is NxNxN Blocks communicate with neighbors through ghost cells

15. Center for Space Environment Modeling http://csem.engin.umich.edu The Octtree Data Structure

16. Center for Space Environment Modeling http://csem.engin.umich.edu Parallel Distribution of the Blocks

17. Center for Space Environment Modeling http://csem.engin.umich.edu Optimized Load Balancing

18. Center for Space Environment Modeling http://csem.engin.umich.edu Parallel Performance

19. Center for Space Environment Modeling http://csem.engin.umich.edu Why Explicit Time-Stepping May Not Be Good Enough Explicit schemes have time step limited by CFL condition: Δt < Δx/fastest wave speed. High Alfvén speeds and/or small cells may lead to smaller time steps than required for accuracy. The problem is particularly acute near planets with strong magnetic fields. Implicit schemes do not have Δt limited by CFL. Explicit schemes have time step limited by CFL condition: Δt < Δx/fastest wave speed. High Alfvén speeds and/or small cells may lead to smaller time steps than required for accuracy. The problem is particularly acute near planets with strong magnetic fields. Implicit schemes do not have Δt limited by CFL.

20. Center for Space Environment Modeling http://csem.engin.umich.edu BDF2 second-order implicit time-stepping scheme requires solution of a large nonlinear system of equations at each time step. Newton linearization allows the nonlinear system to be solved by an iterative process in which large linear systems are solved. Krylov solvers (GMRES, BiCGSTAB) with preconditioning are robust and efficient for solving large linear systems. Schwarz preconditioning allows the process to be done in parallel: Each adaptive block preconditions using local data only MBILU preconditioner BDF2 second-order implicit time-stepping scheme requires solution of a large nonlinear system of equations at each time step. Newton linearization allows the nonlinear system to be solved by an iterative process in which large linear systems are solved. Krylov solvers (GMRES, BiCGSTAB) with preconditioning are robust and efficient for solving large linear systems. Schwarz preconditioning allows the process to be done in parallel: Each adaptive block preconditions using local data only MBILU preconditioner Building a Parallel Implicit Solver

21. Center for Space Environment Modeling http://csem.engin.umich.edu Timing Results Halem = 192 CPU Compaq ES-45 Chapman = 256 CPU SGI 3800 Lomax = 256 CPU Compaq ES-45 Grendel = 118 CPU PC Cluster (1.6 GHz AMD) Halem = 192 CPU Compaq ES-45 Chapman = 256 CPU SGI 3800 Lomax = 256 CPU Compaq ES-45 Grendel = 118 CPU PC Cluster (1.6 GHz AMD)

22. Center for Space Environment Modeling http://csem.engin.umich.edu Fully implicit scheme has no CFL limit, but each iteration is expensive (memory and CPU) Fully explicit is inexpensive for one iteration, but CFL limit may mean a very small Δt Set optimal Δt limited by accuracy requirement: Solve blocks with unrestrictive CFL explicitly Solve blocks with restrictive CFL implicitly Load balance explicit and implicit blocks separately Fully implicit scheme has no CFL limit, but each iteration is expensive (memory and CPU) Fully explicit is inexpensive for one iteration, but CFL limit may mean a very small Δt Set optimal Δt limited by accuracy requirement: Solve blocks with unrestrictive CFL explicitly Solve blocks with restrictive CFL implicitly Load balance explicit and implicit blocks separately Getting the Best of Both Worlds - Partial Implicit

23. Center for Space Environment Modeling http://csem.engin.umich.edu Comparison of Explicit, Implicit and Partial Implicit

24. Center for Space Environment Modeling http://csem.engin.umich.edu Timing Results for Space Weather on Compaq

25. Center for Space Environment Modeling http://csem.engin.umich.edu Controlling the Divergence of B Projection Scheme (Brackbill and Barnes) Solve a Poisson equation to remove div B after each time step. Expensive on a block adaptive parallel grid. 8-Wave Scheme (Powell and Roe) Modify MHD equations for non-zero divergence so it is advected. Simple and robust but div B is not small. Non-conservative terms. Diffusive Control (Dedner et al.) Add terms that diffuse the divergence of the field. Simple but it may diffuse the solution too. Conservative Constrained Transport (Balsara, Dai, Ryu, Tóth) Use staggered grid for the magnetic field to conserve div B Exact but complicated. Does not allow local time stepping. Projection Scheme (Brackbill and Barnes) Solve a Poisson equation to remove div B after each time step. Expensive on a block adaptive parallel grid. 8-Wave Scheme (Powell and Roe) Modify MHD equations for non-zero divergence so it is advected. Simple and robust but div B is not small. Non-conservative terms. Diffusive Control (Dedner et al.) Add terms that diffuse the divergence of the field. Simple but it may diffuse the solution too. Conservative Constrained Transport (Balsara, Dai, Ryu, Tóth) Use staggered grid for the magnetic field to conserve div B Exact but complicated. Does not allow local time stepping.

26. Center for Space Environment Modeling http://csem.engin.umich.edu Effect of Div B Control Scheme

27. Center for Space Environment Modeling http://csem.engin.umich.edu From Codes To Framework The Sun-Earth system consists of many different interconnecting domains that are independently modeled. Each physics domain model is a separate application, which has its own optimal mathematical and numerical representation. Our goal is to integrate models into a flexible software framework. The framework incorporates physics models with minimal changes. The framework can be extended with new components. The performance of a well designed framework can supercede monolithic codes or ad hoc couplings of models.

28. Center for Space Environment Modeling http://csem.engin.umich.edu Why Frameworks? Difficult to create complex science codes, integrating multiple codes interfaces runtime behavior Difficult to modify or extend large systems adding new physics modules updating codes, … Difficult to utilize complete systems what I/O, parameters needed how to submit to multiple sites Framework: A reusable system design,. Component: A packaging of executable software with a well-defined interface Coupling components does not mean the science is correct.

29. Center for Space Environment Modeling http://csem.engin.umich.edu Physics Domains ID Models Solar Corona SCBATSRUS Eruptive Event Generator EEBATSRUS Inner Heliosphere IHBATSRUS Solar Energetic Particles SPKótas SEP model Global Magnetosphere GMBATSRUS Inner Magnetosphere IMRice Convection Model Ionosphere Electrodynamics IERidleys potential solver Upper Atmosphere UAGeneral Ionosphere Thermosphere Model (GITM)

30. Center for Space Environment Modeling http://csem.engin.umich.edu Space Weather Modeling Framework

31. Center for Space Environment Modeling http://csem.engin.umich.edu The SWMF Architecture

32. Center for Space Environment Modeling http://csem.engin.umich.edu Parallel Layout and Execution LAYOUT.in for 20 PE-s SC/IH GM IM/IE ID ROOT LAST STRIDE #COMPONENTMAP SC 0 9 1 IH 0 9 1 GM 10 17 1 IE 18 19 1 IM 19 19 1 #END

33. Center for Space Environment Modeling http://csem.engin.umich.edu Parallel Field Line Tracing Stream line and field line tracing is a common problem in space physics. Two examples: Coupling inner and global magnetosphere models Coupling solar energetic particle model with MHD Tracing a line is an inherently serial procedure Tracing many lines can be parallelized, but Vector field may be distributed over many PE-s Collecting the vector field onto one PE may be too slow and it requires a lot of memory

34. Center for Space Environment Modeling http://csem.engin.umich.edu Coupling Inner and Global Magnetosphere Models Pressure Inner magnetosphere model: needs the field line volumes, average pressure and density along field lines connected to the 2D grid on the ionosphere. Global magnetosphere model: needs the pressure correction along the closed field lines:

35. Center for Space Environment Modeling http://csem.engin.umich.edu Interpolated Tracing Algorithm 1. Trace lines inside blocks starting from faces. 2. Interpolate and communicate mapping. 3. Repeat 2. until the mapping is obtained for all faces. 4. Trace lines inside blocks starting from cell centers. 5. Interpolate mapping to cell centers.

36. Center for Space Environment Modeling http://csem.engin.umich.edu Parallel Algorithm without Interpolation PE 1 PE 3 PE 2 PE 4 2b. If not done send to other PE. 1. Find next local field line. 3. Go to 1. unless time to receive. 6. Go to 1. unless all finished. 2. If there is a local field line then 2a. Integrate in local domain. 4. Receive lines from other PE-s. 5. If received line go to 2a.

37. Center for Space Environment Modeling http://csem.engin.umich.edu Interpolated versus No Interpolation

38. Center for Space Environment Modeling http://csem.engin.umich.edu Set B 0 to a magnetogram based potential field. Obtain MHD steady state solution. Use source terms to model solar wind acceleration and heating so that steady solution matches observed solar wind parameters. Perturb this initial state with a flux rope. Follow CME propagation. Let CME hit the Magnetosphere of the Earth. Set B 0 to a magnetogram based potential field. Obtain MHD steady state solution. Use source terms to model solar wind acceleration and heating so that steady solution matches observed solar wind parameters. Perturb this initial state with a flux rope. Follow CME propagation. Let CME hit the Magnetosphere of the Earth. Modeling a Coronal Mass Ejection

39. Center for Space Environment Modeling http://csem.engin.umich.edu Initial Steady State in the Corona Solar surface is colored with the radial magnetic field. Field lines are colored with the velocity. Flux rope is shown with white field lines. Solar surface is colored with the radial magnetic field. Field lines are colored with the velocity. Flux rope is shown with white field lines.

40. Center for Space Environment Modeling http://csem.engin.umich.edu Close-up of the Added Flux Rope

41. Center for Space Environment Modeling http://csem.engin.umich.edu Two Hours After Eruption in the Solar Corona

42. Center for Space Environment Modeling http://csem.engin.umich.edu 65 Hours After Eruption in the Inner Heliosphere

43. Center for Space Environment Modeling http://csem.engin.umich.edu Sun to Earth CME Simulation In Solar Corona and Heliosphere the resolution ranges from 1/32R S to 4 R S Between 4 and 14 million cells SC/IH grid In Global Magnetosphere the resolution ranges from 1/8R E to 8 R E

44. Center for Space Environment Modeling http://csem.engin.umich.edu The Zoom Movie

45. Center for Space Environment Modeling http://csem.engin.umich.edu What Happens at Earth -More Detail

46. Center for Space Environment Modeling http://csem.engin.umich.edu More Detail at Earth Density and magnetic field at shock arrival time Before shockAfter shock South Turning B Z North Turning B Z Pressure and magnetic field

47. Center for Space Environment Modeling http://csem.engin.umich.edu Ionosphere Electrodynamics Current Potential Before shock hits. After shock: currents and the resulting electric potential increase. Region-2 currents develop. Although region-1 currents are strong, the potential decreases due to the shielding effect.

48. Center for Space Environment Modeling http://csem.engin.umich.edu Upper Atmosphere The Hall conductance is calculated by the Upper Atmosphere component and it is used by the Ionosphere Electrodynamics. After the shock hits the conductance increases in the polar regions due to the electron precipitation. Note that the conductance caused by solar illumination at low latitudes does not change significantly. Before shock arrival After shock arrival

49. Center for Space Environment Modeling http://csem.engin.umich.edu Performance of the SWMF

50. Center for Space Environment Modeling http://csem.engin.umich.edu

51. 2003 Halloween Storm Simulation with GM, IM and IE Components The magnetosphere during the solar storm associated with an X17 solar eruption. Using satellite data for solar wind parameters Solar wind speed: 1800 km/s. Time: October 29, 0730UT Shown are the last closed field lines shaded with the thermal pressure. The cut planes are shaded with the values of the electric current density.

52. Center for Space Environment Modeling http://csem.engin.umich.edu GM, IM, IE Run vs. Observations

53. Center for Space Environment Modeling http://csem.engin.umich.edu The Space Weather Modeling Framework (SWMF) uses sate-of-the-art methods to achieve flexible and efficient coupling and execution of the physics models. Missing pieces for space weather prediction: Better models for solar wind heating and acceleration; Better understanding of CME initiation; More observational data to constrain the model; Even faster computers and improved algorithms. The Space Weather Modeling Framework (SWMF) uses sate-of-the-art methods to achieve flexible and efficient coupling and execution of the physics models. Missing pieces for space weather prediction: Better models for solar wind heating and acceleration; Better understanding of CME initiation; More observational data to constrain the model; Even faster computers and improved algorithms. Concluding Remarks