1. Computation Fluid Dynamics & Modern Computers: Tacoma Narrows Bridge Case Study Farzin Shakib ACUSIM Software, Inc. 2003 SGI Technical Users ’ Conference June 13, 2003

2. Content  Brief Company & Product Overview  Tacoma Narrows Bridge Upgrade Project Effectiveness of CFD in a large scale project Need for ever larger and more complex solutions  Design for High Performance Computers  Conclusions

3. Company Overview  Founded in 1994  First product sold in 1997 Who has continued in production and steadily increased usage  Marketed through distributor channels and direct sales  Success stories: Visteon employs AcuSolve for climate control (ICCE) and underhood cooling (UPV) analyses of production cars ORCA ™ : A fully integrated mixing package for chemical and pharmaceutical companies

4. CFD Product: AcuSolve  A powerful general-purpose finite element incompressible flow solver  AcuSolve ’s differentiation: Robustness Most problems solved on the first attempt Speed Coupled solver on distributed parallel machines Accuracy Highly accurate in space and time while globally and locally conservative Features Rich set of functionality; continuously growing  An ideal enabling technology for integrated engineering and scientific applications

5. Tacoma Narrows Bridge Upgrade Project Advanced Simulation Bechtel National, Inc.

6. CFD Project  Use CFD to predict time-varying loads and moments on new bridge caissons (East and West) for Flood and Ebb flows Show CFD can predict known drag coefficients Show CFD can match experiment scale model  Use CFD to predict variation in loads due to changing flow directions for different caisson drafts for full scale geometry

7. Validation: flow around standard objects

8. Cylinder Test Case - Re= 1.15 e+08  Time history of computed Drag

9. Cylinder Test Case - Performance  Mesh: 500K Tet-elements, 100K nodes  2-processors 900MHz Itanium II  Steady-state RANS Solution (SA turbulence model): 50 time steps in 30 minutes CD = 0.5  Transient Hybrid RANS/LES Solution (DES): 1000 time steps in 8 hours CD = 0.7  Same solution with 3.5M Tet-element mesh

10. Square Cylinder Test Case - Re = 115000  Time history of computed Drag NACA at Re=100000: Cd=2.0, Parker &Welsh: Sh= 0.12 CFD fine mesh: Cd~ 1.98, Sh~ 0.12

11. Computational Domain / Mesh 3M Tet/Prism-elements ICEM-CFD Tetra Module Transient simulation Hybrid RANS/LES model

12. East Side Caisson – 4.6 m/s Flood

13. Force Time History – East Side – 4.6 m/s Flood

14. East/West caissons – 3.6 m/s Ebb East Side West Side Flow streamlines around caisson

15. East/West caissons – 3.6 m/s Ebb East Side West Side Velocity profile at 120 ft depth

16. Validation Against Experiment  HR Wallingford (in UK) conducted experiment Re = 10^5 – Model-Scale  CFD Solutions at Re = 10^8 – Full-Scale Re = 10^5 – Model-Scale  Comparison CFD Re=10^5 compared with experiment CFD Re=10^5 / Re=10^8 comparison is used to Scale up HRW measurements to Re=10^8

17. Model Scale vs. Full Scale – Y Force

18. Frequency Spectrum HRW Results CFD Results  t =0.3s

19. Observations  Excellent agreement between CFD and Experiment on large scale measures Total Drag and Lift Drag and Lift RMS (total energy) Primary shedding frequency  CFD solution is too coarse to resolve fine scale frequencies

20. Fine Mesh  Mesh: 23M Tet-elements – 5M nodes 8x finer than previous meshes  Hybrid RANS/LES transient solution  Requires fast parallel machine  32-processor SGI Altix (900 MHz Itanium II) 750 time steps 30 hours 8.5 GBytes of memory

21. Fine Mesh Solution

22. Fine & Regular Mesh Solution Comparison

23. Demand on CFD  Users are placing ever larger demands on performance of CFD solvers  AcuSolve responds through: Accurate numerical technology –Galerkin/Least-Squares Finite Element technology Fast/robust linear and nonlinear solution algorithms –Coupled pressure/velocity iterative solution algorithms Design for High Performance Computers –Single processor performance –Parallel performance

24. Single-Processor Performance  Fast element formation Block “ elements ” in self similar groups Arrange memory with stride 1 block data Optimize memory layout for cache access  Fast linear solution Access LHS matrices sequentially Reorder equations based on proximity Avoid excessive indirections  Hybrid programming C for overall control of program Fortran for numerical computation

25. Parallel Processing  Designed from beginning for coarse grain parallel machines Host-less architecture  Domain decomposition distributes elements and nodes to different processors Transparent to the user The number of processors may be changed at restart MPI for distributed-memory machines MPI and/or OpenMP for SMP machines  All algorithms work seamlessly on parallel machines

26. Conclusions  AcuSolve accurately computed transient turbulent flows around caisson and pier.  Solution was used to design anchoring system of new span of the Tacoma Narrows Bridge.  Solution agreed well with experiment on all primary features.  Fine scale features required finer meshes, which required faster and larger machines. This trend will continue in the foreseeable future.