Adaptive Multiscale Simulation Infrastructure - AMSI  Overview: o Industry Standards o AMSI Goals and Overview o AMSI Implementation o Supported Soft Tissue Simulation o Results W.R. Tobin, D. Fovargue, D. Ibanez, M.S. Shephard Scientific Computation Research Center Rensselaer Polytechnic Institute

Current Industry Standards – Physical Simulations  Overwhelming majority of numerical simulations conducted in HPC (and elsewhere) are single scale o Continuum (e.g. Finite Element, Finite Difference) o Discrete (e.g. Molecular Dynamics)  Phenomena at multiple scales can have profound effects on the eventual solution to a problem (e.g. fine-scale anisotropies) 2

Current Industry Standards – Physical Simulations  Typically a physical model or scale is simulated using a Single Program Multiple Data (SPMD) style of parallelism o Quantities of interest (mesh, tensor fields, etc.) distributed across the parallel execution space 3 Geometric Model Partition Model Distributed Mesh

Current Industry Standards – Physical Simulations  Interacting physical models and scales introduce a much more complex set of requirements in our use of the parallel execution space o Writing a new SPMD code for each new multiscale simulation would require intense reworking of legacy codes used for single-scale simulations (possibly many times over)  Need approach which can leverage the work that has gone into creating and perfecting legacy simulations in the context of massively parallel simulations with interacting physical models 4 primary spmd code auxiliary spmd code auxiliary spmd code auxiliary spmd code

AMSI Goals  Take advantage of proven legacy codes to address the needs of multimodel problems o Minimize need to rework legacy codes to execute in more dynamic parallel environment o Only desired edit/interaction points are those locations in the code where the values produced by multiscale interactions are needed.  Allow dynamic scale load-balancing and process scale reassignment to reduce process idle time when a scale is blocked or underutilized 5

AMSI Goals  Hierarchy of focuses o Abstract-Level: Support for implementing multi-model simulations on massively parallel HPC machines o Simulation-Level: Allow dynamic runtime workflow management to implement versatile adaptive simulations o Theory-Level: Provide generic control algorithms (and hooks to allow specialization) supported by real-time minimal simulation meta- modeling o Developer-Level: Facilitate all of the above while minimizing AMSI system overheads and maintaining robust code 6 simulation goals physics analysis scale/physics linking models scale/physics linking models physical attributes simulation initialization simulation state control adaptive simulation control discretization, model, linking error estimates discretization, model, linking error estimates discretization, model, scale linking improvement model hierarchy control limits based on measured parameters

AMSI Goals  Variety of end-users targeted o Application Experts: Simulation end-users who want answers to various problems o Modeling Experts: Introduce codes expressing new physical models Combine proven physical models in new ways to describe multiscale behavior o Computational Experts: Introduce new discretization methods Introduce new numerical solution methods Develop new parallel algorithms 7

AMSI Overview  General meta-modeling services o Support for modeling computational scale-linking operations and data Model of scale-tasks and task-relations denoting multiscale data transfer o Specializing this support will facilitate interaction with high- level control and decision-making algorithms 8 explicit and computational domains math and computational models explicit and computational tensor fields scaleX explicit and computational domains math and computational models explicit and computational tensor fields scaleY geometric interactions model relationships field transformations scale linking

AMSI Overview  Dynamic management of the parallel execution space  Process reassignment will use load balancing support for underlying SPMD distributed data as well as the implementation of state-specific entry/exit vectors for scale-tasks. o Load balancing support of scale-coupling data is supported by the meta model of that data in the parallel space o Other data requires support for dynamic load balancing in any underlying libraries o Can be thought of as a hierarchy of load-balancing operations Multiple scale-task communication/computation balancing Single scale-task load balancing (standard SPMD load balancing operators) 9

AMSI Implementation  AMSI::ControlService o Primary application interaction point for AMSI, tracks the overall state of the simulation. o Higher-level control decisions use this object to implement those decisions and update the simulation meta-model.  AMSI::TaskManager o Maintain the computational meta-model of the parallel execution space and various simulation models.  AMSI::RelationManager o Manage computational scale-linking communication and load balancing required for dynamic management of parallel execution space. 10

AMSI Implementation  Real-time minimal simulation meta-model o Initialization actions Scale-tasks and their scale-linking relations o Runtime actions Data distributions representing discrete units of generic scale-linking data Communication patterns determining distribution of scale linking communication down to individual data distribution units o Shift to more dynamic scale management will require new control data to be reconciled across processes and scales Change initialization actions to be (allowable) runtime actions 11 Initialization Runtime scaleX scaleY scaleZ scaleX scale-linking data scaleY communication patterns

AMSI Implementation  Two forms of control data parallel communication o Assembly is a scale-task collective process. o Reconciliation is collective on the union of two scale-tasks associated by a communication relation. 12 scaleX scaleY Assembly scaleX scaleY Reconciliation

AMSI Implementation  Scale linking communication patterns o Constructed via standard distribution algorithms, or o Hooks provided for user-implemented pattern construction, unique to each data distribution 13 CommPatternAlgo_Register(relation_id, CommPatternCreate_FuncPtr); CommPattern_Create(dataDist_id, owner_scale_id, foreign_scale_id); scaleX scaleY

AMSI Implementation  Scale linking communication is handled, on both sides, via a single function call o Determines whether the process belongs to the sending or recving scale-task o Communicates scale-linking quantities guided by a communication pattern o Buffer is contiguous memory segment packed with POD data, MPI_Datatype must describe that datatype o At present a data distribution is limited to one POD representation 14 Communicate(relation_id, pattern_id, buffer, MPI_Datatype);

AMSI Implementation  Shift to phased communication and dynamic scale-task management will introduce new requirements o Will reduce number of explicit control data reconciliations o Will require the introduction of implicit control data reconciliations during scale-linking operations Primary simulation control points 15 scaleX scaleY assemble reconcile communicate

AMSI Implementation  Shift to phased communication and dynamic scale-task management will introduce new requirements o Will reduce number of explicit control data reconciliations o Will require the introduction of implicit control data reconciliations during scale-linking operations Primary simulation control points 16 scaleX scaleY assemble reconcile / communicate compute

Biotissue  Multiscale soft-tissue mechanics simulation o Engineering Scale : Macroscale (Finite Element Analysis) o Fine Scale controlling engineering scale behavior: Microscale Fiber-Only-RVE (Quasistatics) Microscale Fiber-Matrix-RVE (FEA) (future project) Additional cellular scale(s) (FEA)  Intermediate scale between current scales o Scale linking Deformations to RVE Force/displacement to the engineering scale 17 Macroscale Fiber-Only

Biotissue Implementation  Scalable implementation with parallelized scale-tasks 18 Macroscale Microscale macro0 macro1macro2 macroN micro0micro1 micro2microM-1microM

Biotissue Implementation  Scalable implementation with parallelized scale-tasks 19 Macroscale Microscale macro0 macro1macro2 macroN micro0micro1 micro2microM-1microM

Biotissue Implementation  Scalable implementation with parallelized scale-tasks o Ratio of macroscale mesh elements per macroscale process to number of microscale processes determines neighborhood of scale-linking communication 20 Macroscale Microscale

Biotissue Implementation o Macroscale - Parallel Finite Element Analysis Distributed partitioned mesh, distributed tensor fields defined over the mesh, distributed linear algebraic system Stress field values characterize macro-micro ratio o Fiber-only microscale - Quasistatics code ~1k Nodes per RVE Rapid assembly and solve times per RVE in serial implementation Strong scaling with respect to macroscale mesh size Initial results use fiber-only at every macroscale integration point to generate stress field values o Fiber-matrix microscale – Parallel FEA Order of magnitude more nodes per RVE (~10k-40k) More complex linear system assembly and longer solve times (nonlinear) necessitate parallel implementation per RVE 21

Biotissue Implementation  Incorporating fiber-and-matrix microscale RVEs o Hierarchy of parallelism Macroscale SPMD code Microscale fiber-only code Microscale fiber-matrix SPMD code o Nonlinear problem o Macroscale to auxiliary scales relation more complex Constitutive relation Fiber-only RVE Fiber-matrix RVE o Adaptive processes allow these relations to change over time  Intermediate cellular scale will introduce even further complexities to this situation 22

Results 23  Biotissue simulation was run with a test problem o Standard tensile test macroscale geometry (dogBone) o Various discretizations of the geometry Current results for 20k and 200k, working on memory issues (microscale) for 2m elements and higher o Holding macroscale count fixed, varying microscale o Holding micrsocale count fixed, varying macroscale o Varying both scales

Results 24

Results 25 Time (s) # of processes on microscale 1 st iteration of multiscale solver 20k mesh – 2 macro processes

Results 26 Time % # of processes on microscale 1 st iteration of multiscale solver 20k mesh – 2 macro processes

Results  (varying macroscale while holding micro fixed) 27 Time (s) # of processes on macroscale 1 st iteration of multiscale solver 200k mesh – 7680 macro processes

Results 28 Arrows indicate increasing macro size (4,8,16,32,64) Communication 1 st iteration of multiscale solver 200k mesh – time ratios Time % # of processes on microscale

Results 29  (weak scaling results) Time (s) # of processes on macroscale

Closing Remarks 30  Results are just starting to come out of the implementation o Need to identify critical areas of each scale code to improve overall performance of multiscale code o Shift to phased communication will allow macroscale to process microscale results as they arrive, increasing computation communication overlap o Contributing microscale code needs memory footprint improvements to mitigate running out of memory during longer runs (larger meshes)