Computer Science Overview © Laxmikant Kale Department of Computer Science June 5, 2001 ©2000 Board of Trustees of the University of Illinois 1

CS Faculty and Staff Investigators T. Baker M. Bhandarkar M. Campbell E. de Sturler H. Edelsbrunner R. Fiedler M. Heath J. Hoeflinger L. Kale J. Liesen J. Norris D. Padua D. Reed P. Saylor K. Seamons A. Sheffer S. Teng M. Winslett plus numerous students

Computer Science Research Overview Computational Mathematics and Geometry Linear solvers and preconditioners Eigensolvers Mesh generation and adaptation Interface propagation and interpolation Computational Environment Software integration framework Parallel I/O and data migration Performance tools and techniques Computational steering Visualization

Linear Solvers Analysis of Krylov subspace methods Development of faster and more robust Krylov subspace methods Development of more robust methods for ill-conditioned linear and nonlinear systems Improvement of Jacobi-Davidson methods for eigenvalue problems Derivation of sharper error estimates and stopping criteria for iterative methods Preconditioners for radiation transport problems

Mesh Generation and Adaptation Mesh adjustment for moving boundaries Data structures for non-conformal meshes in discontinuous Galerkin methods Space-time mesh generation Model simplification for meshing Surface parameterization and element shape improvement Skin model for evolving boundary space-time meshing

Mesh Generation and Adaptation Library for mixed 3D cohesive element meshes a program for introducing cohesive elements based on material types. Interaction with Geubelle 2) Mesh quality measures & Laplace smoothing in the ALE code with Mike Brandyberry 3) Continuing research on Space-Time meshing in 2DxTIME 4) Surface parameterization with E. de Sturler's group in colaboration with Sandia.

Software Integration Framework Flexible framework for coupling stand-alone application codes Encapsulation via objects and threads Runtime environment to support dynamic behavior (e.g., refinement, load balancing) Intelligent interface for mediating communication between component modules

Roccom -- Component Manager Mechanism for inter-component data and function sharing Roccom API Programming interface for application modules Roccom developers interface C++ interface for service modules Roccom implementations Runtime systems of Roccom

Rationales of Roccom Object-oriented philosophy Enforce encapsulation of data Enable runtime polymorphism of functions Minimal changes to existing applications Each component manages its own data, and Publicize data and functions by registering to Roccom Maximal concurrency in code development No need to worry about details of others data structure Maximal flexibility for integration Switching application component, service component, and runtime system with minimal changes to codes

Data and Function Organization Window – distributed object Geometrically, portion of mesh in contact with another More generally, collection of interface data and functions Pane – chuck of distributed object Portion of window specific to thread with its own arrays Each thread can have multiple panes Attribute – public data member of window Window attr., pane attr., node attr., and element attr. Composite attr.: e.g., “mesh”, “all” Function – public function member of window

Status of Roccom Roccom implementation Complete base implementation for SPMD style integration Charm++ and Autopilot based are ongoing Current Roccom service components Rocface-2.0 – data transfer between nonmatching meshes Interface convergence check Two switchable HDF output modules One sequential, and one parallel using Panda Application modules Rocflo, Rocsolid, Rocfrac, and Rocburn

Rocface –Interface Component Robust and efficient algorithm for overlaying two surface meshes + = Rocface handles transferring data between nonmatching surface meshes. To handle the data transfer, Rocface first constructs a reference mesh, which is the overlay, I.e., the common refinement of two meshes.

Example Overlay on Star Grain Example of overlaying two meshes on a star grain geometry. It demonstrates the ability of Rocface of handling relatively complex geometry with sharp edges and corners.

Least Squares Data Transfer Minimizes error and enforces conservation Handles node and element centered data Made possible by the overlay Achieved superb experimental results After computing the overlay, Rocface transfers data using a least-squares formulation. This formulation is both accurate and conservative, and works for both node and element Centered data. This formulation can be solved accurately and efficiently using the overlay of two meshes. Our experimental results show that this method works much better than others. The figures Show the displacements using two different methods for a burning cavity problem with Uniform pressure and uniform regression after 500 time steps. The figure to the left is Using the new method, which is very accurate. The one to the right used the conservative load transfer algorithm by Farhat. It is apparent that the one to the left is much better than The one to the left. (The right one had an error of about 20%.) Load transfer (Farhat) Our method

Performance of GEN1 Using Charm++

Load Balancing with Charm++ 267.75 299.85 301.56 235.19 Time Step 133.76 149.01 150.08 117.16 Pre-Cor Iter 46.83 52.20 52.50 41.86 Solid update 86.89 96.73 97.50 75.24 Fluid update 8P3,8P2 w. LB 8P3,8P2 w/o LB 16P2 16P3 Phase

New Capabilities Shrink and expand FEM Framework Based on Charm++ the set of assigned processors can be changed at runtime FEM Framework Fortan 90 and C++ based parallelization of unstructured mesh-based codes (FEM) Components can be used for generation of communication-lists: used for new ROCCrack Planned: insertion and deletion of elements Based on Charm++

AMPI What is AMPI: Adaptive load balancing for MPI Programmers Uses Charm++’s load balancing framework Uses multiple MPI threads per processor light weight threads Recent progress: Compiler support for automatic conversion global variables packing-unpacking functions Cross communicators Allows multiple components to communicate across Two independent MPI “Worlds” can communicate Implemented for ROCFLO/ROCSOLID separation Picture

AMPI and ROC* Rocflo Rocface Rocsolid

Plans for the framework Automatic out-of-core execution Take advantage of Data-driven execution Cluster Management Job Scheduler to maximize throughput Using Strechable jobs (as well as fixed-size ones)

Parallel I/O and Data Migration Parallel output of snapshots for GEN1 Combine arrays for different blocks into single virtual array Output multiple arrays at once using array group Manage metadata for outputting HDF files for Rocketeer Automatic tuning of parallel I/O performance Data migration concurrent with application Automatic choice of data migration strategy Alpha testing and benchmarking of Panda 3.0

Autopilot Interfacing Library AP Roccom Implementation Library for interfacing multiple applications over a network For parallel multi-physics or multi-component scientific applications. Little to no source code changes required C, C++, F77, F90, HPF Independant of application parallelization implementation Cross platform interfacing with Linux, AIX, Solaris, and Irix Built on Globus

Autopilot ROCCOM Provides mechanisms for runtime computational steering Requires some (little) source code changes Mechanisms for user/client based or automatic steering Dynamic starting, stopping, and swapping of application components at runtime Provides mechanisms for runtime performance tuning and visualization Built on top of existing Pablo performance suite Mechanisms for automatic performance based steering at runtime Remote performance visualization on workstations or I-desk using Virtue

Visualization with Rocketeer Built on Visualization Toolkit (VTk) and OpenGL Supports Structured and unstructured grids Cell-centered and node-centered data Ghost cells Seamless merging of multiple data files Automated animation Smart HDF reader Translucent isosurfaces New features – spheres, etc Parallel, client-server implementation in progress

Department of Computer Science Prof. Laxmikant Kale Department of Computer Science University of Illinois at Urbana-Champaign 2262 Digital Computer Laboratory 1304 West Springfield Avenue Urbana, IL 61801 USA kale@cs.uiuc.edu http://www.cs.uiuc.edu/contacts/ faculty/heath.html telephone: 217-333-6268 fax: 217-333-1910