1. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Finite Element Solution of PDEs Christoph Zenger Nadine Dieminger, Frank Günther, Wolfgang Herder, Andreas Krahnke, MiriamMehl, Tobias Neckel, Markus Pögl, Markus Langlotz, Tobias Weinzierl Institut für Informatik TU München

2. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Problem: Numerical solution of partial differential equations by the finite element method Numerical kernel: Computation of the product: discrete operator A · approx. solution u Desired properties: Multilevel scheme adaptive efficient on modern computer architectures Parallel with good load balance Complex geometries

3. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Concepts: Hierarchical structures Informatics: Stacks and trees Geometry: space trees Numerics: Hierarchical bases and generating systems Mathematics: Space filling curves

4. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Informatics Stack Binary tree 1 2 3 1 2 3 4 5 6 10 54 6 7 9 811 13 151412

5. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Ternary tree 1 2 35 4 6 79 8 10 1113 12

6. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Geometry: quadtree ternary space tree

7. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Dimension recursive construction ternary space tree: d steps in d dimensions instead of one

8. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Hierarcal structures in Numerics Hierarchical basis and generating system

9. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

10. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

11. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Ternary hierarchical basis ternary generating system

12. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Mathematics: Space filling curves Basic template (Hilbert): Recursive construction:

13. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Peano curve (dimension recursive): Basic template: Recursive construction: Works for arbitrary dimension

14. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Space-Trees and Element- Oriented Operator Evaluation i,j i+1,j i,j-1 i-1,j i,j+1 1 1-4 1 1

15. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Space-Trees and Element- Oriented Operator Evaluation i,j i+1,j i,j-1 i-1,j i,j+1 ½ -1 ½

16. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Space-Trees and Element- Oriented Operator Evaluation i,j i+1,j i,j-1 i-1,j i,j+1 -1 ½ ½

17. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Space-Trees and Element- Oriented Operator Evaluation i,j i+1,j i,j-1 i-1,j i,j+1 ½ -1 ½

18. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Space-Trees and Element- Oriented Operator Evaluation i,j i+1,j i,j-1 i-1,j i,j+1 ½ ½ -1

19. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Concept of stacks:

20. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Space-Trees and Space-Filling Curves ordering of cells along the Peano-curve line-stacks with alternating linear (locally deterministic) processing order

21. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Adaptive Space-Trees and Space-Filling Curves adaptive grids, generating systems hiding of points on different levels additional colours, point stacks 8 stacks (independent of refinement depth)

22. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Locality

23. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Locality: Length of a cache line : m bytes Length of solution vector: s bytes Minimal number of cache misses: n min = s/m. Actual number of cache misses: n = 1.1*n min. Memory efficiency Essentially only solution data are stored Definition of domain and refinement structure: only 2 bits per degree of freedom! ( unknowns on a PC for Laplace equation)

24. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München 3D-Poisson-equation on a cube

25. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München More complicated Domains

26. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Adaptivity for Complicated Geometries arbitrary refinements automatic boundary detection

27. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

28. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

29. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

30. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München tau-extrapolation 2nd orderextrapolation Hh||e|| L 2 ||e|| 1 ||e|| L 2 ||e|| 1 3 -2 3 -3 3,310 -3 3,810 -3 1,610 -3 1,810 -3 3 -3 3 -4 3,310 -4 3,810 -4 2,210 -5 2,510 -5 3 -4 3 -5 3,510 -5 4,410 -5 1,610 -7 3,310 -7

31. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Adaptivity + Full Multigrid fourth order solution for the actual grid refinement (hierarchical surplus, tau, dual approach) additive v-cycles with tau-extrapolation

32. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Adaptivity + Full Multigrid fourth order solution for the actual grid refinement (hierarchical surplus, tau, dual approach) additive v-cycles with tau-extrapolation

33. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

34. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Adaptive refinement

35. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

36. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Parallelization – Partitioning Using the Peano-Curve process 1 process 2

37. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Parallelization – Communication process 1 process 2

38. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

39. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

40. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

41. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

42. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Results – Speedup / Efficiency Poisson equation –Sphere geometry –Static non-regular grid –# dof: 23,118,848 # cells: 26,329,806 –Myrinet cluster # processesT (all)T (comm.)Parallel Speedup Parallel Efficiency 13155.18011 21614.865.371.950.976 4845.8026.533.730.932 8460.4927.486.850.856 16243.8222.7412.930.809

43. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Continuity preservingFE-Scheme for the Navier-Stokes equation

44. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Time dependant 2D-Navier- Stokes-Equation Reynoldsnumber 2

45. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Time dependant 2D-Navier- Stokes-Equation Reynoldsnumber 1000

46. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Re = 100, 729*81 grid points, velocity(right) pressure (left)

47. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Conclusion Navier-Stokes fluid-structure interactions diffusion equation with non-constant coefficients financial pricing enhanced boundary treatment

48. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München Thank You !

49. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München

50. Cache-Optimal Parallel Solution of PDEs Ch. Zenger Informatik V, TU München