12 March 2004 A High-Fidelity, High-Performance Turbulent Dispersion Framework Damian Rouson Combustion Modeling & Scaling U.S. Naval Research Laboratory, Washington, DC Prof. Joel Koplik & Karla Morris Depts. of Physics & Mechanical Engineering The City College of CUNY, New York, NY Sponsors: NSF, NRL

Overview  Motivation & Objectives  Previous work  Physics –Fire suppression –Quantum turbulence  Software –Componentization strategy –O.O. Fortran 90/95 Implementation  Future Work –Study architecture  complexity  performance –Study fine-grained parallelism (OpenMP)

MotivationMotivationObjectives Previous Work PhysicsSoftware Future Work NObjects 0 Point masses: particles, droplets 1 Quantum vortices, “snails” 2 Interfaces (fluid/fluid & fluid/solid) 3 Two-fluid mixtures  A broad class of phenomena involve interactions between fluid turbulence & N-dimensional objects  Algorithms & data structures for these problems exhibit a high degree of commonality.  Exploiting software design commonalities can lead to greater code reuse and more efficient interoperation

 To build an extensible set of software components suitable for the N={0,1} subset of turbulent dispersion problems.  To quantify the influence of object-oriented design & componentization on software complexity and performance.  To simulate new physics & improve engineering models: –Fire suppression –Quantum turbulence –Boundary-layer combustion ObjectivesMotivationObjectives Previous Work PhysicsSoftwareResults Conclusions & Future Work

Previous Work MotivationObjectives PhysicsSoftware Future Work  Rouson & Eaton (2001) “On the preferential concentration of solid particles in a fully-developed turbulent channel flow”, J. Fluid Mech.  Rouson & Xiong (2004) “Design metrics in quantum turbulence simulations: How physics influences software architecture”, Scientific Programming.  Decyk, Norton & Szymanski (1997-) Object-oriented programming in Fortran 90, ACM Fortran Forum, etc.  Akin (2003), Object-Oriented Programming via Fortran 90/95, Cambridge U. Press.  Armstrong et al. (1999-) “Toward a common component architecture for high-performance scientific computing”

Particle-Laden Flow Direct Numerical Simulation (DNS) of Navier-Stokes Equations: Spectral Spatial Representation: Lagrangian Particle Equation of Motion: 3 rd -Order Runge-Kutta Time AdvancementMotivationObjectives Previous Work PhysicsSoftware Future Work Isotropic turbulence:

Particle-Laden Flow Particles near centerplane of a channel flow: Preferential concentration:  Particles move independently from each other and slip relative to flow  Coherent flow structures segregate particles according to their inertia –e.g. “cyclone separation” by vortices MotivationObjectives Previous Work PhysicsSoftware Future Work

Below 2.17 K, liquid helium behaves as a two-fluid mixture:  Normal viscous fluid  Inviscid superfluid – characterized by discrete (quantized) vortex motions – vortex core ~ 1 A across – can be modeled by 1. Nonlinear Schrodginger Eq. 2. Euler equation 3. Discrete vortex equations  Surprisingly, turbulence statistics match those of classical flows. match those of classical flows. Quantum Turbulence Density ratio of each fluid: MotivationObjectives Previous Work PhysicsSoftware Future Work

Superfluid: vortex filament method. Quantum Turbulence Arbitrary origin Increasing  Vortex filament Normal fluid: DNS of Navier-Stokes equations. Material properties:  ’  Quantum of circulation:  = ħ/m He = 9.97*10 -4 cm 2 /sec Equations of motion: (Biot-Savart Law) MotivationObjectives Previous Work PhysicsSoftware Future Work

Animation Barenghi et al. (1997) Our result A=B=C=0.748cm/s, 0.2cm As a prelude to turbulence, we studied the steady Arnold-Beltrami-Childress (ABC) flow: Quantum Turbulence MotivationObjectives Previous Work PhysicsSoftware Future Work

ComponentizationFluid Provides: d_dt(), u(x) d_dt(), u(x) Uses: f(x) f(x) VectorField Grid CloudProvides: d_dt(), f(x) d_dt(), f(x) Uses: u(x) u(x) Droplet TangleProvides: d_dt(), f(x) d_dt(), f(x) Uses: u(x) u(x) VortexPoint IntegratorProvides: euler euler RK3 RK3 Uses: d_dt() d_dt()MotivationObjectives Previous Work PhysicsSoftware Future Work

OOP in Fortran 90/95 OOP Concept Fortran Construct ClassMODULE Abstract data type Derived type InheritanceAggregation Polymorphism Generic interfaces MotivationObjectives Previous Work PhysicsSoftware Future Work

Fortran Implementation MODULE Cloud_Class USE Droplet_Class USE Droplet_Class PRIVATE PRIVATE PUBLIC :: d_dt (), operator(*), operator(-) PUBLIC :: d_dt (), operator(*), operator(-) TYPE Cloud TYPE Cloud PRIVATE PRIVATE TYPE(Droplet), DIMENSION(:), ALLOCATABLE :: drop TYPE(Droplet), DIMENSION(:), ALLOCATABLE :: drop END TYPE Cloud END TYPE Cloud INTERFACE d_dt INTERFACE d_dt MODULE PROCEDURE d_dt_Cloud MODULE PROCEDURE d_dt_Cloud END INTERFACE END INTERFACE CONTAINS CONTAINS FUNCTION d_dt_Cloud(this) RESULT(dState_dt) FUNCTION d_dt_Cloud(this) RESULT(dState_dt) TYPE(Cloud), INTENT(IN) :: this TYPE(Cloud), INTENT(IN) :: this TYPE(Cloud), POINTER :: dState_dt TYPE(Cloud), POINTER :: dState_dt DO i=1,n DO i=1,n dState_dt%drop(i) = d_dt(this%drop(i)) dState_dt%drop(i) = d_dt(this%drop(i)) END DO END DO Inheritance Polymorphism EncapsulationMotivationObjectives Previous Work PhysicsSoftware Future Work

Fortran Implementation MODULE Droplet_Class PRIVATE PRIVATE PUBLIC :: d_dt(), operator(*), operator(-) PUBLIC :: d_dt(), operator(*), operator(-) TYPE Droplet TYPE Droplet PRIVATE PRIVATE REAL :: St REAL :: St REAL, DIMENSION(3) :: s REAL, DIMENSION(3) :: s REAL, DIMENSION(3) :: v REAL, DIMENSION(3) :: v END TYPE Droplet END TYPE Droplet … CONTAINS CONTAINS FUNCTION d_dt_Droplet(this,u) RESULT(dState_dt) FUNCTION d_dt_Droplet(this,u) RESULT(dState_dt) TYPE(Droplet),POINTER :: dState_dt TYPE(Droplet),POINTER :: dState_dt TYPE(Droplet),INTENT(IN) :: this TYPE(Droplet),INTENT(IN) :: this REAL,INTENT(IN),DIMENSION(3) :: u REAL,INTENT(IN),DIMENSION(3) :: u … dState_dt%St = 0.0 dState_dt%St = 0.0 dState_dt%s = this%v dState_dt%s = this%v dState_dt%v = (u-this%v)/this%St dState_dt%v = (u-this%v)/this%St MotivationObjectives Previous Work PhysicsSoftware Future Work

Fortran Implementation MODULE Integrator_Class USE Cloud_Class USE Cloud_Class USE Fluid_Class USE Fluid_Class PRIVATE PRIVATE PUBLIC :: euler_Integrator PUBLIC :: euler_Integrator TYPE Integrand TYPE Integrand PRIVATE PRIVATE TYPE(Cloud), POINTER :: cloud_ptr TYPE(Cloud), POINTER :: cloud_ptr TYPE(Fluid), POINTER :: fluid_ptr TYPE(Fluid), POINTER :: fluid_ptr END TYPE Integrand END TYPE Integrand… CONTAINS CONTAINS SUBROUTINE euler_Integrator(this,dt) SUBROUTINE euler_Integrator(this,dt) TYPE(Integrand), INTENT(INOUT) :: this TYPE(Integrand), INTENT(INOUT) :: this REAL(NBYTES) :: dt REAL(NBYTES) :: dt IF (ASSOCIATED(this%fluid_ptr) ) THEN IF (ASSOCIATED(this%fluid_ptr) ) THEN this%fluid_ptr = this%fluid_ptr + dt*d_dt(this%fluid_ptr) ELSE IF (ASSOCIATED(this%cloud_ptr)) THEN ELSE IF (ASSOCIATED(this%cloud_ptr)) THEN…MotivationObjectives Previous Work PhysicsSoftware Future Work Run-time polymorphism via dynamic dispatching

 Complete componentization of F90 code  Study performance via fine-grained parallelization –Several recent developments presage a potentially renewed importance of OpenMP:  Multi-core chips (Intel)  Hardware support for multithreading (Sun’s Niagara)  Efficient emulation of SMP via NUMA clusters (SGI’s Altix) Future Work MotivationObjectives Previous Work PhysicsSoftware Future Work