1. Computer Science Overview
Laxmikant Kale
Department of Computer Science
June 5, 2001
©2000 Board of Trustees of the University of Illinois 1

2. CS Faculty and Staff Investigators
T. Baker
M. Bhandarkar
M. Campbell
E. de Sturler
H. Edelsbrunner
R. Fiedler
M. Heath
J. Hoeflinger
L. Kale
J. Liesen
J. Norris
D. Padua
D. Reed
P. Saylor
K. Seamons
A. Sheffer
S. Teng
M. Winslett
plus numerous students

3. Computer Science Research Overview
Computational Mathematics and Geometry
Linear solvers and preconditioners
Eigensolvers
Mesh generation and adaptation
Interface propagation and interpolation
Computational Environment
Software integration framework
Parallel I/O and data migration
Performance tools and techniques
Computational steering
Visualization

4. Linear Solvers Analysis of Krylov subspace methods
Development of faster and more robust Krylov subspace methods
Development of more robust methods for ill-conditioned linear and nonlinear systems
Improvement of Jacobi-Davidson methods for eigenvalue problems
Derivation of sharper error estimates and stopping criteria for iterative methods
Preconditioners for radiation transport problems

5. Mesh Generation and Adaptation
Mesh adjustment for moving boundaries
Data structures for non-conformal meshes in discontinuous Galerkin methods
Space-time mesh generation
Model simplification for meshing
Surface parameterization and element shape improvement
Skin model for evolving boundary
space-time meshing

6. Mesh Generation and Adaptation
Library for mixed 3D cohesive element meshes
a program for introducing cohesive elements based on material types.
Interaction with Geubelle
2) Mesh quality measures & Laplace smoothing in the ALE code
with Mike Brandyberry
3) Continuing research on Space-Time meshing in 2DxTIME
4) Surface parameterization
with E. de Sturler's group
in colaboration with Sandia.

7. Software Integration Framework
Flexible framework for coupling stand-alone application codes
Encapsulation via objects and threads
Runtime environment to support dynamic behavior (e.g., refinement, load balancing)
Intelligent interface for mediating communication between component modules

8. Roccom -- Component Manager
Mechanism for inter-component data and function sharing
Roccom API
Programming interface for application modules
Roccom developers interface
C++ interface for service modules
Roccom implementations
Runtime systems of Roccom

9. Rationales of Roccom Object-oriented philosophy
Enforce encapsulation of data
Enable runtime polymorphism of functions
Minimal changes to existing applications
Each component manages its own data, and
Publicize data and functions by registering to Roccom
Maximal concurrency in code development
No need to worry about details of others data structure
Maximal flexibility for integration
Switching application component, service component, and runtime system with minimal changes to codes

10. Data and Function Organization
Window – distributed object
Geometrically, portion of mesh in contact with another
More generally, collection of interface data and functions
Pane – chuck of distributed object
Portion of window specific to thread with its own arrays
Each thread can have multiple panes
Attribute – public data member of window
Window attr., pane attr., node attr., and element attr.
Composite attr.: e.g., “mesh”, “all”
Function – public function member of window

11. Status of Roccom Roccom implementation
Complete base implementation for SPMD style integration
Charm++ and Autopilot based are ongoing
Current Roccom service components
Rocface-2.0 – data transfer between nonmatching meshes
Interface convergence check
Two switchable HDF output modules
One sequential, and one parallel using Panda
Application modules
Rocflo, Rocsolid, Rocfrac, and Rocburn

12. Rocface –Interface Component
Robust and efficient algorithm for overlaying two surface meshes + =
Rocface handles transferring data between nonmatching surface meshes.
To handle the data transfer, Rocface first constructs a reference mesh, which is the overlay, I.e., the common refinement of two meshes.

13. Example Overlay on Star Grain
Example of overlaying two meshes on a star grain geometry. It demonstrates the ability of Rocface of handling relatively complex geometry with sharp edges and corners.

14. Least Squares Data Transfer
Minimizes error and enforces conservation
Handles node and element centered data
Made possible by the overlay
Achieved superb experimental results
After computing the overlay, Rocface transfers data using a least-squares formulation.
This formulation is both accurate and conservative, and works for both node and element
Centered data.
This formulation can be solved accurately and efficiently using the overlay of two meshes.
Our experimental results show that this method works much better than others. The figures
Show the displacements using two different methods for a burning cavity problem with
Uniform pressure and uniform regression after 500 time steps. The figure to the left is
Using the new method, which is very accurate. The one to the right used the conservative
load transfer algorithm by Farhat. It is apparent that the one to the left is much better than
The one to the left. (The right one had an error of about 20%.)
Load transfer (Farhat)
Our method

15. Performance of GEN1 Using Charm++

16. Load Balancing with Charm++
267.75
299.85
301.56
235.19
Time Step
133.76
149.01
150.08
117.16
Pre-Cor Iter
46.83
52.20
52.50
41.86
Solid update
86.89
96.73
97.50
75.24
Fluid update
8P3,8P2 w. LB
8P3,8P2 w/o LB
16P2
16P3
Phase

17. New Capabilities Shrink and expand FEM Framework Based on Charm++
the set of assigned processors can be changed at runtime
FEM Framework
Fortan 90 and C++ based parallelization of unstructured mesh-based codes (FEM)
Components can be used for generation of communication-lists: used for new ROCCrack
Planned: insertion and deletion of elements
Based on Charm++

18. AMPI What is AMPI: Adaptive load balancing for MPI Programmers
Uses Charm++’s load balancing framework
Uses multiple MPI threads per processor
light weight threads
Recent progress:
Compiler support for automatic conversion
global variables
packing-unpacking functions
Cross communicators
Allows multiple components to communicate across
Two independent MPI “Worlds” can communicate
Implemented for ROCFLO/ROCSOLID separation
Picture

19. AMPI and ROC*
Rocflo
Rocface
Rocsolid

20. Plans for the framework
Automatic out-of-core execution
Take advantage of Data-driven execution
Cluster Management
Job Scheduler to maximize throughput
Using Strechable jobs (as well as fixed-size ones)

21. Parallel I/O and Data Migration
Parallel output of snapshots for GEN1
Combine arrays for different blocks into single virtual array
Output multiple arrays at once using array group
Manage metadata for outputting HDF files for Rocketeer
Automatic tuning of parallel I/O performance
Data migration concurrent with application
Automatic choice of data migration strategy
Alpha testing and benchmarking of Panda 3.0

22. Autopilot Interfacing Library
AP Roccom Implementation
Library for interfacing multiple applications over a network
For parallel multi-physics or multi-component scientific applications.
Little to no source code changes required
C, C++, F77, F90, HPF
Independant of application parallelization implementation
Cross platform interfacing with Linux, AIX, Solaris, and Irix
Built on Globus

23. Autopilot ROCCOM
Provides mechanisms for runtime computational steering
Requires some (little) source code changes
Mechanisms for user/client based or automatic steering
Dynamic starting, stopping, and swapping of application components at runtime
Provides mechanisms for runtime performance tuning and visualization
Built on top of existing Pablo performance suite
Mechanisms for automatic performance based steering at runtime
Remote performance visualization on workstations or I-desk using Virtue

24. Visualization with Rocketeer
Built on Visualization Toolkit (VTk) and OpenGL
Supports
Structured and unstructured grids
Cell-centered and node-centered data
Ghost cells
Seamless merging of multiple data files
Automated animation
Smart HDF reader
Translucent isosurfaces
New features – spheres, etc
Parallel, client-server implementation in progress

25. Department of Computer Science
Prof. Laxmikant Kale
Department of Computer Science
University of Illinois at Urbana-Champaign
2262 Digital Computer Laboratory
1304 West Springfield Avenue
Urbana, IL USA
faculty/heath.html
telephone:
fax: