1. High Performance Computing for MHD Ramakanth Munipalli, P.-Y. Huang HyPerComp Inc., Westlake Village, CA 91361 In collaboration with S.Smolentsev, N.B.Morley, A.Ying, G.Pulugundla, M.Abdou, UCLA 2 nd EU-US DCLL Workshop, UCLA, November 15, 2014 Research supported by current and prior SBIR funding from DOE

2. Outline 1. HyPerComp Inc. – who are we? Small company, located 30 miles NW of LA, estd. 1998, Lengthy, fruitful collaboration with UCLA in numerical MHD 2. MHD – what have we done so far in fusion related MHD modeling? 3. Multiphysical extensions Mass transport, corrosion, integrated multiphysical modeling 4. What are we working on now? 5. High performance computing – some tools and techniques Some prospects and future directions

3. 1. HyPerComp Inc. – Who are we?

4. HyPerComp develops high performance computing software in fluid mechanics and electromagnetics 3 Recent Examples: Very large scale CPU/GPU parallel electromagnetics modeling Liquid rocket combustion dynamics Coupled fluid-structure-acoustics modeling for rotorcraft

5. Computational Electromagnetics RCS, SAR Imagery and range profiles, Penetrable materials, Antennas, Optics Magnetohydrodynamics & Plasmadynamics Flow Control, Liquid metal MHD – nuclear fusion, Combustion control Computational Fluid Dynamics Incompressible to Hypersonic flow, Turbulent Combustion, Rotorcraft, Acoustics, Morphing structures, Optimization High Performance Multiphysical Computations Very high order accurate solutions, CPU/GPU based solvers, Reduced Basis Models A Technology Portfolio TEMPUS HYCE

6. 2. MHD – what have we done so far? MHDTop-level concern in fusion blanket studies (pressure drop, corrosion, tritium transport, etc.) Difficult problem to solve No analytical solutions in all but the most elementary flows Numerical solutions riddled with difficulties and uncertainties

7. HyPerComp Incompressible MHD solver for Arbitrary Geometry HIMAG was developed by HyPerComp in a research partnership with UCLA to model the flow of liquid metals in nuclear fusion reactor design. These flows are characterized by very high magnetic field strength, complex geometry and fluid-solid coupling via electric current and heat. HIMAG is a pioneer in the reliable computation of such flows among both commercial as well as research simulation software.

8.  HIMAG is a parallel, unstructured mesh based MHD solver.  High accuracy at high Hartmann numbers is maintained even on non- orthogonal meshes  HIMAG can model single-phase as well as two-phase (free surface) flows  Multiple conducting solid walls may be present in the computational domain  Graphical User Interfaces are provided for the full execution of HIMAG  Heat transfer, natural convection, temperature dependent properties can be modeled  Extensive validation and benchmarking has been performed for canonical problems. Cases involving Ha > 1000 have never been demonstrated on non-rectangular meshes prior to HIMAG HIMAG: Technical Summary

9. Governing Equations Two phase fluid flow without MHD Two phase fluid flow with MHD, heat transfer

10. Treatment of electromagnetic fields in liquid metal MHD When the magnetic Reynolds number is low (Re m = μ 0 σLU << 1), induced magnetic field is negligible. Applied magnetic field is the total magnetic field, and an electric potential may be derived: The potential is obtained from the condition that the current density is solenoidal Ohm’s law: A more general treatment solves for the induced magnetic field: solid fluid Electric current can flow across interfaces between media Appropriate simplifications (J=0, etc.), or, Boundary conditions (normal current = 0) can be used

11. Numerical difficulties in computing high Ha MHD flows 1.The need to resolve and minimize numerical errors in Hartmann layers (Thickness about 1/Ha) 2. The pace of convergence of Poisson equation solvers (for Pressure, Electric potential, divergence of B) 3.Computationally intensive corrections for non-orthogonal meshes 4.Long periods of integration needed to account for flow development, unsteady effects 5.Time taken to develop CAD to CFD mesh for high Ha problems The high Hartmann number problem As Ha = increases, regions of sharp variations in velocity and electric current density appear in the flow. Numerically, small inaccuracies in the Hartmann layer are greatly amplified Flow enters a strong B-field

12. 1 / Ha 1 / sqrt(Ha) U(z) j(y,z)  Ha ≈Ha -1 B  // ≈Ha -1/2 2h Hartmann layer Side layer FLOW The exacting needs of numerical MHD

13. Fully developed flow at Ha = 10,000 in a duct with square cross section – compared with the exact analytical solution Verification against canonical benchmark problems Fully developed flow at Ha = 1,000 in a duct with circular cross section

14. Ra = 10 3 Ra = 10 4 Ra = 10 5 Ra = 10 6 RaMeshNu HIMAG Nu comp 10 3 21x21 41x41 81x81 1.1227 1.1191 1.11811.118 10 4 21x21 41x41 81x81 2.3084 2.2611 2.24882.245 10 5 21x21 41x41 81x81 4.9370 4.6312 4.54904.522 10 6 21x21 41x41 81x81 161x161 10.661 9.4598 8.9885 8.8659 8.829 Natural Convection A good match of Nusselt number with published data has been observed. Second order accuracy has been ascertained.

15. Ha=5800 Validation against experimental data: ALEX duct experiment (1987, ANL) Circular duct Square duct

16. Wall functions Contact resistance Coarse mesh in Ha layers Jump in potential at arbitrary material interfaces can now be captured HIMAG: Some MHD specific numerics

17. Robust modeling of strong natural convection Newton-Krylov based schemes are used to perform matrix inversion in a upwind semi-implicit procedure to stabilize the simulation of flow with strong natural convection (Gr = 10 9 ). 2a=20 cm B 2m2m Y X 40 cm Gr = 10 9 in a square cavity Streamlines (left) and isotherms (right) Flow in a 3-D channel with MHD and heat transfer (results in next chart)

18. Upward flow# cellsSpeedup Full solution iwall=0299,440 1 Wall function iwall=1070,080 19.18 Wall function iwall=11162,3366.2 Flow Re = 10,000, Ha = 400, Gr = 10 7 Wall functions are used to model MHD flow in an insulating channel with dimensions similar to DCLL. These functions are applied at all walls (iwall=10) or only at Hartmann walls (iwall=11) Temperature Velocity contours Full MHD solution (left) and solution with wall functions (right) g A case study in the use of wall functions in solution acceleration Sample speedup results on 16 CPUs Comparison of full and wall function solutions along centerline

19. DCLL concept and crucial MHD features

20. z x y B g Full 3-D MHD (with natural convection) model of the DCLL Flow enters from lower right, exits above A low conductivity flow channel insert is present Velocity profiles shown on right and pressure on left

21. 3. Multiphysical studies

22. Fusion relevant liquid metal MHD includes a variety of multiphysical phenomena  Fluid, heat and mass transport  Natural convection  Steady and unsteady electromagnetic phenomena  Contact resistance (thermal and electric)  Ferromagnetic effects  Two phase flow, surface tension  Phase change (due to high heat flux, pressure variation, etc.)  Corrosion, electrochemistry at walls  Fluid-structure interaction

23. HIMAG: Free surface flow simulations MTOR, experiment at UCLA Liquid metal jet in a magnetic field

24. Plasma Liquid metal Plasma-liquid interaction Plasma-liquid metal interaction in a magnetic field: DiMES HIMAG uses the level set method in unstructured meshes. Method permits large deformations of the free surface. We are presently working on massively parallel free surface capture simulations with scalable adaptive meshing (w. RPI) HIMAG: Free surface flow simulations – contd.

25. Mass Transport  Tritium transport We built an independent software named CATRIS from the basic data structure of HIMAG, to focus on specific issues in mass transport.  Corrosion Some capabilities to simulate corrosion were added to CATRIS, together with Lagrangian models for particulate transport

26. CATRIS (Corrosion And TRItium transport Solver) Written as a new stand-alone system, CATRIS focuses on the following: (a)the transport of tritium and its permeation through walls (b)corrosion and deposition of iron contained within structural materials of the system.

27. A sample study to inject particles in an MHD flow, which migrate in response to the gradient in an applied magnetic field. These particles will be produced at walls using a corrosion BC and deposited likewise. Tritium concentration in module computed for B=4T (Ha = 15 000), u=0.065m/s. Tritium transport and particle transport models in CATRIS

28. 4. What are we working on now?

29. Current Research Objectives  Brand new software implementation of the induced magnetic field formulation – adds ability to model high magnetic Reynolds number, strongly unsteady flows  Improvement in time accuracy  Enhanced robustness and speed – better parallelization  Dramatically reduce simulation time for MHD flows in geometries of practical interest – DCLL, etc.  Transition to general EM applications in materials processing

30. Induced magnetic field formulation

31. b) Velocity profile along x=0 and z=0 sections Case 1 – 3-D Lid-driven cavity with conducting walls except top lid (Re=100, Ha=10, C w =0.4) a) Schematic view of 3-D lid-driven cavity with conducting walls (Blue area), Red – Liquid. U B

32. f) Convergence history. Case 1 – Cavity with conducting walls - Continued e) Comparison of velocity curves by B-formulation with those by potential solver (HIMAG).

33. Case 2 – 3-D Lid-driven cavity with insulating walls (Re=100, Ha=45) b) Velocity profile along x=0.5 and y=0.5 sections a) Schematic view of 3-D lid- driven cavity with insulating walls: Red – Liquid. U B

34. Comparison of velocity u(z) by B- formulation with published data Case 2 – Cavity with insulating walls - Continued Comparison of velocity w(x) by B- formulation with published data

35. 5. High Performance Computing

36. Mathematical methods Multigrid methods: Agglomeration, unnested pre- meshed scheme, algebraic multigrid Hybrid meshes Implicit schemes – time stepping, accuracy CN, AB, SIRK schemes SIMPLE scheme for steady flow Local time stepping Full matrix solvers – BICGSTAB on CPU/GPU Interpolation procedures Data storage: Non-orthogonal correction GPU-based programming Rapid prototyping Canonical decomposition of DCLL geometry User interfaces for inlet, fci, bend, etc. Template for multigrid (3-D) – dir. agglomeration Template for hybrid mesh generation blocking Applications of analysis EM coupling between neighboring channels Fringing fields, self-consistent field formulations Wall functions – insulating, perfectly conducting Wall functions for fringing fields Patched analytical – numerical solution Combination of duct flow solutions Summary of our approach

37. Poisson solver with Neumann BCs Hunt’s fully developed channel flow 3-D circular duct Ha = 10, Re = 10, c w = 0.1 3-D rectangular duct Ha = 0, Re = 15,250 3-D rectangular duct Ha = 100, Re = 10, c w = 0.1 Poisson solver acceleration by sparse matrix storage and inversion

38. The use of non-orthogonal meshes Among the many benefits of the HIMAG approach has been the preservation of accuracy on arbitrary mesh systems – making the calculation more efficient Hybrid/unstructured meshes reduce number of mesh points needed, by transitioning between regions of different resolution requirements

39. “Template” driven mesh generation

40. HIMAG: A nested multigrid method A sequence of unnested grids for multigrid Poisson solution, showing reduced convergence time (above), automatic generation of a sequence of meshes from CAD input (below)

41. Some prospects and future directions

42. Summary We have a number of ongoing software development activities which can serve simulation needs in fusion: Robust, comprehensive, self-consistent MHD physics modeling CPU/GPU parallel computing Highly scalable parallel mesh adaptation Strengths in coupled MHD/heat/mass transfer analysis (numerous existing physical and numerical models) Integrated and customizable software solutions

43. Future Directions EM toolkit for flow and materials processing 1.Combined calculation of fluid flow, heat transfer (radiative, convective), mass transfer (including corrosion) and electromagnetics - applications in aerospace propulsion, materials processing 2. Localized plasma models – thermal as well as weak ionization, extended to large length scales as source terms (external flow control, plasma assisted processing, radar analysis) 3. Two phase flow - models of free surfaces such as melt layers 4. Phase change - evaporation, melting, solidification, crystallization, etc.