1. Parallelization of 2D Lid-Driven Cavity Flow
Asif Salahuddin
Ahmad Sharif
Jens Kehne
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

2. Parallelization of 2D Lid-Driven Cavity Flow
Objectives
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

3. Parallelization of 2D Lid-Driven Cavity Flow
Our objectives
Numerical simulation of fluid dynamics, using the Lattice-Boltzmann method
Parallelize the code using MPI
Study speedup and scalability
Allow to run large problem sizes in reasonable time
Allow to run them at all, for that matter (memory requirements)
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

4. Parallelization of 2D Lid-Driven Cavity Flow
Concept
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

5. The Lattice-Boltzmann method
The Lattice-Boltzmann equation:
Velocity directions:
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

6. Parallelization of 2D Lid-Driven Cavity Flow
Top-down vs. bottom-up
Partial
differential
equations
(Navier-Stokes)
Partial
differential
equations
(Navier-Stokes)
Discretization
Multi-scale analysis
Difference
equations
(Conserved Quantities?)
Discrete model (LGCA or LBM)
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

7. Parallelization of 2D Lid-Driven Cavity Flow
Fluid nodes
The entire problem is represented as a grid of fluid nodes
Fluid nodes hold velocities towards all neighbors
New grid state computed for discrete time steps
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

8. Parallelization of 2D Lid-Driven Cavity Flow
Wall bounceback
The fluid domain is surrounded by walls
On each timestep, the direction of links hitting a wall is reversed
Walls may be moving
Changes the momentum of the fluid close to it
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

9. Parallelization of 2D Lid-Driven Cavity Flow
Implementation
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

10. Parallelization of 2D Lid-Driven Cavity Flow
Domain decomposition
Each processor processes part of the grid
: Ghost nodes
Represent border nodes of the neighbors
: Border nodes
Updated by neighbors
: Inner nodes
We can update these alone
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

11. Automatic decomposition
Factorize and merge
Factorize x and y dimension and #procs
Divide x and y by prime factors of #procs
Goal: Try to keep the processor’s grids as square as possible
Best relation between inner and border nodes
Minimizes communication
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

12. Automatic decomposition - demo30
# CPUs: 6 20
#CPUs: 6 = 2 * 3 X-axis: 30 = 2 * 3 * 5 Y-axis: 20 = 2 * 2 * 5 10 10
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

13. Parallelization of 2D Lid-Driven Cavity Flow
Optimizations
Overlapping wall bounceback and communication
About 5% speedup
Overlapping inner node computation with communication
Massive slowdown!
Probably due to cache effects
Making use of regular communication pattern
Slower (we have no idea why!)
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

14. Parallelization of 2D Lid-Driven Cavity Flow
Experimental results
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

15. Parallelization of 2D Lid-Driven Cavity Flow
Experimental setup
Lonestar Linux University of Texas
Part of the Teragrid project
1300 compute nodes
2 Intel Xeon 2.66 GHz dual-core CPUs per node
42.6 GFLOPS/node
8GB RAM/node
Linux kernel 2.6, 64 bit
Infiniband interconnect, fat tree topology
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

16. Parallelization of 2D Lid-Driven Cavity Flow
Actual speedup
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

17. Relation to expected speedup
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow

18. Parallelization of 2D Lid-Driven Cavity Flow
Questions
04 December 2008
Parallelization of 2D Lid-Driven Cavity Flow