The MicroGrid: A Scientific Tool for Modeling Grids Andrew A. Chien SAIC Chair Professor Department of Computer Science and Engineering University of California, San Diego April 30, 2001

Andrew A. Chien – GrADS Site Visit (4/01)2 Outline Motivation What is a MicroGrid? Validating Models Status Future Work

Andrew A. Chien – GrADS Site Visit (4/01)3 Motivation Need tools to study complex dynamic Grid behavior »complex non-linear dynamic behavior »Tightly couple communication, computing, and storage resources »Performance, Availability, Failure Complementary approaches useful, but insufficient »MacroGrids –Limitations of scale and actual configuration –Major logistical efforts »Other Simulations –Network-only (internet/networking) –Application level (simple resource models) »Enable design of robust, reliable, good performing Grids and Grid applications

Andrew A. Chien – GrADS Site Visit (4/01)4 Grid Application Developer How will my software behave on the projected hardware configuration? (performance) How will it behave dynamically? (robustness) How will it interact with other Grid applications an uses of the system? How can I make this a robust, stable, reusable application? “Cactus” “Zeus-MP” “Netsolve” “SF-Express” “Distributed Viz” “GTomo” “Tardis”

Andrew A. Chien – GrADS Site Visit (4/01)5 Grid System Software Developer Libraries – network, performance instrumentation, runtime environment (e.g. Globus) Program Preparation System – dynamic compilers, runtime, etc. Do these things work and how well? With what applications and what range of applications? “GrADS” “PPS” “Nimrod” “Globus” “NWS” Grid Researchers

Andrew A. Chien – GrADS Site Visit (4/01)6 Grid System Administrator What if I change my resource access policies? What if I add/take away these resources? What if I change the “price” charged for resources? What happened to my Grid when it melted down last week?

Andrew A. Chien – GrADS Site Visit (4/01)7 MicroGrid Goals Runtime environment for GrADS experiments (a la MacroGrid) Develop technology and tools to support specialized Grid communities (a la MacroGrid) Realistic modelling of a broad range of Grid systems, applications, environments, and dynamic behavior »Execution of real applications (tools and middleware) »Scale to large experiments »High fidelity simulation, support variety of speed + fidelity –Network, compute, memory, disk »Observable, repeatable behavior

Andrew A. Chien – GrADS Site Visit (4/01)8 Outline Motivation What is a MicroGrid? Validating Models Status Future Work

Andrew A. Chien – GrADS Site Visit (4/01)9 Grid Application Virtual Grid MicroGrid Software LAN Workgroup Scalable Cluster Heterogeneous Environment MicroGrid Modeling A scientific tool for modeling Computational Grids » Run arbitrary Grid applications on any virtual Grid resources » Allow the study of complex dynamic behavior of large systems

Andrew A. Chien – GrADS Site Visit (4/01)10 MicroGrid Today Processor speed modeling Memory size modeling Virtualized Resource description (GIS/MDS) Network Virtualization Online Network Simulation => runs the Globus software => runs Globus applications on a Linux/Alpha testbed

Andrew A. Chien – GrADS Site Visit (4/01)11 Using a MicroGrid Find some physical resources Configure a Virtual Grid Submit a Globus Job to it Observe Execution (which occurs in virtual time) DeConfigure the Virtual Grid Grid Application Virtual Grid, “MicroGrid” MicroGrid Software

Andrew A. Chien – GrADS Site Visit (4/01)12 Outline Motivation What is a MicroGrid? Validating Models Status Future Work

Andrew A. Chien – GrADS Site Visit (4/01)13 MicroGrid Validation Simulate an benchmarks and applications various Grid systems Run simulations on the physical hardware Compare to published results

Andrew A. Chien – GrADS Site Visit (4/01)14 Validation on Micro-benchmarks Memory Capacity Modeling Processor Speed Modeling NSE Network Modeling Each resource model is validated

Andrew A. Chien – GrADS Site Visit (4/01)15 Validation on NPB Benchmarks Comparison to published cluster NPB results »Set parameters based on known published relative resource performance -- processor and network performance »Alpha cluster (Alpha’s + 100Mbit Ethernet) and HPVM cluster Overall execution time matches within 4%

Andrew A. Chien – GrADS Site Visit (4/01)16 NPB over WAN vBNS A fictional Cluster Varying WAN bandwidth

Andrew A. Chien – GrADS Site Visit (4/01)17 NPB over WAN (Cont.) No background network traffic Performance is insensitive to network bandwidth Shows a simulation of hypothetical cluster on WAN

Andrew A. Chien – GrADS Site Visit (4/01)18 Internal Behavior of NPB Autopilot tools for Program Tracing (in MicroGrid environment) Traces from MicroGrid and real Grid Match within 5%

Andrew A. Chien – GrADS Site Visit (4/01)19 Validation on Large Applications Cactus PDE Solver Framework on Alpha cluster WaveToy program, various Matrix sizes Execution time matches within 7%

Andrew A. Chien – GrADS Site Visit (4/01)20 Outline Motivation What is a MicroGrid? Validating Models Status Future Work

Andrew A. Chien – GrADS Site Visit (4/01)21 MicroGrid Today Uses Globus Supports Globus applications and tools Incorporates models for »Processor speed »Memory capacity »Virtualized Resource Description (GIS/MDS) »Network Virtualization »Online Network Simulation Used via standard submission interfaces Not yet available for external users, improving robustness and adding modules

Andrew A. Chien – GrADS Site Visit (4/01)22 What have we learned? Demonstrated accurate simulation of Grid environments and applications Demonstrated ability to support existing applications and tools (critical for significant experiments) Existing network simulation tools are inadequate Existing network traffic models are inadequate Deriving network configuration information is challenging Extrapolation of results is a major challenge due to nonlinearity of behavior

Andrew A. Chien – GrADS Site Visit (4/01)23 What have we learned? (cont) There’s a LOT more work to be done to support »large-scale, high speed simulations, »with flexible choice of resource models, »simulating a wide range of environments (config, background activity, etc.), and »executing on a wide range of physical hardware resources.

Andrew A. Chien – GrADS Site Visit (4/01)24 Milestones Year 1: Develop Initial Version of MicroGrid toolkit Empirical study of application behavior based on MicroGrid toolkit Year 2: GrADS runtime environment and applications on the MicroGrid (in progress)

Andrew A. Chien – GrADS Site Visit (4/01)25 Outline Motivation What is a MicroGrid? Validating Models Status Future Work

Andrew A. Chien – GrADS Site Visit (4/01)26 Ongoing and Future Activities System Development (Better MicroGrid) »Scalable On-line Network simulation – Xin “Paff” Liu »Variable speed simulation (efficiency) – Ranjita Bhagwan »Network Traffic Modeling (background & coupled load) – Xianan Zhang »Disk Speed Modeling (I/O intensive workloads) – Huaxia Xia Other current activities (Validation, Software) »Scalapack modeling – Match GrADS results »Cactus modeling – Match GrADS results »Porting to x86 Linux »Robustify and package for external release

Andrew A. Chien – GrADS Site Visit (4/01)27 Summary Demonstrated that MicroGrid approach can produce accurate results in modeling »Grid applications »Grid infrastructures »Dynamic behavior Working software Significant validation »Micro-benchmarks; Full benchmarks; Applications … Need to get MicroGrid software to the next level of capability …

Andrew A. Chien – GrADS Site Visit (4/01)28 MicroGrid Team Dr. Andrew Chien (PI) Graduate Students: »Xin “Paff” Liu, Ranjita Bhagwan, Xianan Zhang, Huaxia Xia Former: »Dr. Hyo Jung Song (Postdoc) »Dr. Kenjiro Taura (U Tokyo Professor) »Dennis Jakobsen (MS) For more information see » »

Andrew A. Chien – GrADS Site Visit (4/01)29