1. Parallel Mesh Refinement with Optimal Load Balancing Jean-Francois Remacle, Joseph E. Flaherty and Mark. S. Shephard Scientific Computation Research Center

2. Scope of the presentation The Discontinuous Galerkin Method (DGM) –Discontinuous Finite Elements –Spatial discretization –Time discretization –DG for general conservation laws Adaptive parallel software –Adaptivity –Parallel Algorithm Oriented Mesh Datastructure

3. The DGM for Conservation Laws Find such that Weighted residuals + integration by parts Spatial discretization

4. The DGM for Conservation Laws Discontinuous approximations Conservation on every element

5. The DGM for Conservation Laws Numerical Flux Choices for numerical fluxes –Lax Friedrichs –Roe linearization with entropy fix –Exact 1D Riemann solution (more expensive) Monotonicity not guaranteed –Higher-order limiters

6. Higher order equations Discontinuous approximations needs regularization for gradients

7. Computing higher order derivatives How to compute when u is not even C 0 ? Stable gradients : find such that Or weakly

8. Computing higher order derivatives –Solution of the weak problem: w=u. Weak derivatives are equal, then fields are equal. –If we choose a constrained space for w with no average jumps on interfaces i.e. with –We have –With –And

9. Computing higher order derivatives –For higher order derivatives:

10. Time discretization Explicit time stepping –Efficient in case of shock tracking e.g. Method of lines may be too restrictive due to –Mesh adaptation (shock tracking) –Real geometry's (small features) Local time stepping, use local CFL –The key is the implementation –Important issues in parallel

11. Local time stepping

12. Grouping elements

13. Example: muzzle break mesh Speedup around 50

14. Parallel Issues Good practice in parallel –Balance the load between processors –Minimize communications/computations –Alternate communications and computations Local time stepping –Elementary load depends on local CFL –Not the mostly critical issue

15. Parallel issues Example, load is balanced when –Proc 0 : 2000(1  dt) + 1000 (2  dt) –Proc 1 : 3000(1  dt) + 500 (2  dt) – Total Load : 4000  dt If synchronization after every sub-time steps –Proc 0 waits 1000  dt at the first sync. –Proc 1 waits 1000  dt at the first sync –Maximum parallel speedup = 4/3

16. Parallel issues Solution –Synchronization only after the goal time step –Non blocking sends and receive after each sub- time step –Inter-processor faces store the whole history –Some elements may be “retarded”

17. A Parallel Algorithm Oriented Datastructure

18. Objectives of PAOMD Distributed mesh –Partition boundaries treated like model boundaries –On processor : serial mesh Services –Round of communication –Parallel adaptivity –Dynamic load balancing

19. Dynamic load balancing

20. Example

21. 2D Rayleigh Taylor

22. Four contacts

24. Higher order equations Navier-Stokes –Von Karman vortices –Re = 200 Numerics –use of p=3 –no limiting –filtering In parallel

25. 64 processors of the PSC alpha cluster 1  10 6 to 2.0  10 7 dof’s 128 processors of Blue Horizon 10 8 dof’s Large scale computations

26. Muzzle break problem Process –Input: ProE CAD file –MeshSim: Mesh gen. –Add surface mesh for force computations –Choice of parameters Orders of magnitude –1 day (single proc., no adaptation, LTS) –will need ~ 100 procs. for adaptive computations

27. Force computations Conservation law Integral of fluxes Numerical issues –Geometric search

28. 2D computations Importance of –adaptivity –2nd order method Influence of the muzzle

29. 2D computations

30. 3D computations Challenging –Large number of dof’s –Complex geometry

31. Discussion The issue –Large scale computations –Explicit time stepping, –Load balancing In progress –Semi implicit and implicit schemes –Higher order limiters improved