1. MicroGrid Update & A Synthetic Grid Resource Generator Xin Liu, Yang-suk Kee, Andrew Chien Department of Computer Science and Engineering Center for Networked Systems University of California, San Diego September 13-14, 2004 VGrADS Workshop University of Tennessee, Knoxville

2. MicroGrid Update 9/13/20042 Grid Application Virtual Grid, “MicroGrid” MicroGrid Software LAN Workgroup Scalable Cluster Heterogeneous Environment MicroGrid Enables Deep Study of Grid Dynamics

3. MicroGrid Update 9/13/20043 MicroGrid Highlights Binary Interception enables Transparent Virtualization (SC2000) »“Virtual time” enables wide range of relative performance experiments Scalable Packet-level Simulation provides accurate protocol behavior (SC2003) »Profile and Topology driven Graph Partitioners »Full TCP, Router, OSPF, and BGP modeling MicroGrid validated on diverse benchmarks & grid applications (JOGC 2004) Large-Scale ISP Simulation (20,000+ routers) based on new hierarchical load-balance and partition (SC2004) Source Releases: 2/2003, 7/2003, Version 2.4.4. Feb 2004

4. MicroGrid Update 9/13/20044 Hierarchical Grouping and Load Balance For Further Scalability Scalability Challenges: »Non-linear effect of Minimal Link Latency –Lead to a small MLL »Poor Partition Results from METIS Solution: »Reduce the original graph by merging nodes with link latency less than a threshold The Key: »How to set the link latency threshold

5. MicroGrid Update 9/13/20045 Partition Efficiency (E) E= Es * Ec Es: efficiency decided by achieved MLL »Es = (MLL – Cn)/MLL »Cn: synchronization cost for n physical nodes Ec: efficiency decided by the load balance »Ec = Caverage/Cmax  Tradeoffs between Parallelism and Efficiency  Evaluate Different Partition Outputs w/o Running the Simulation

6. MicroGrid Update 9/13/20046 Scalable Simulation with Hierarchical Load Balance Hierarchical and Profile-based Partition/Balance enables high efficiency Scalable to 20,000 routers using 90 nodes on TeraGrid ~ 40% load balance, ~50% simulation time improvement ~45% parallel efficiency

7. MicroGrid Update 9/13/20047 High Fidelity Modeling Realistic Routing Structure »Flat networks of 20,000 routers (OSPF) »Hierarchical networks of 100 AS’s with 200 routers each (BGP & OSPF) BGP Configuration »Simulate Typical BGP Configuration Practice »Providers, Customers, Peering, Exporting »Realistic Structure »=> ongoing study of how real, open problem in networking community

8. MicroGrid Update 9/13/20048 Applications of MicroGrid Detailed Application Performance and Configuration Studies Resource Discovery and Selection Studies Scheduling Studies Adaptation (Rescheduling) Studies Desktop Grid (P2P Computing) Dynamic Resource Management and Trading Policies Application Deployment Studies Resource Configuration Studies … and Many More …

9. MicroGrid Update 9/13/20049 Challenging Example: Denial of Service User Edge Proxy Proxy Resource Pool (IP Network) Host Overlay Network Application Attacker

10. MicroGrid Update 9/13/200410 Challenges Large Traffics »Malicious attacks »Large overlay network Accuracy »Application latency/ throughput »Packet drop, Link congestion Flexibility »Various network topology »Effect of different overlay structure

11. MicroGrid Update 9/13/200411 Experiment Configuration Overlay Network » ForwardEngine Prototype Application »Apache server / Siege http clients Attackers »simTrinoo UDP traffic ~1000 routers, 64 node overlay network, 200 siege clients, 10GByte/s attacking traffic

12. MicroGrid Update 9/13/200412 Ongoing Experiments Application Performance »With and Without Attacking Traffic »Effect of Overlay Network Placement Resilience of DOS Attack »App Performance under Different Attack Magnitudes Client Failover Policy »How to achieve best performance under attack –Fix Binding –Random Selection –Best Achievable

13. MicroGrid Update 9/13/200413 Detailed Application Behavior All Details: Buffer filling, Packet Drop

14. MicroGrid Update 9/13/200414 Effect of Overlay Placement 2ms68ms3ms 22ms48ms 2ms48ms3ms22ms 67ms Baseline Case1 Case2 Case3 3ms App Server Proxy Nodes App Client

15. MicroGrid Update 9/13/200415 User Response Time Proxy can Reduce the Response Time! The Last Proxy should be Close to the App Server

16. MicroGrid Update 9/13/200416 User Bandwidth Proxy can also Increase the Throughput!

17. MicroGrid Update 9/13/200417 CDF of User Observed Response Time

18. MicroGrid Update 9/13/200418 Summary MicroGrid continues to improve in capability Integrated »Topology Generators (maBrite) »Background Traffic Generators »Automatic Profile-based Load Balance »BGP Configuration Scalable, Large-network Simulations are being achieved at detailed packet-level with Large-scale Parallel Resources

19. yskee@csag.ucsd.edu Synthetic Resource Generator for Computational Grids Yang-suk Kee and Andrew Chien Department of Computer Science and Engineering Center for Networked Systems University of California, San Diego [Kee&Chien, SC2004]

20. 20 15 Needs of Synthetic Platform Generator Evaluation of Resource monitoring system Resource management technique Resource description language Resource selection(matching) algorithm Application scheduling algorithm How to get representative resource configuration scenarios? It is not feasible to build large reconfigurable Grids

21. 21 15 Synthetic Platform Generator Is A tool to synthesize platform configurations for computation Grids by statistical analysis of distributions of existing grid resources Characteristics New models for Grid resource configuration 10,179 processors from 114 clusters Validation of the resource models 10,073 processors from 191 clusters Extrapolation to future Grid resource configuration

22. 22 15 Resources of Interest Commodity-based clusters Processor architecture Processor clock speed Processor cache # of processors per node Memory size per node Disk capacity per node # of nodes per cluster System Area Network (SAN) per cluster

23. 23 15 Resource Distribution: Processor Architecture Samples 1yr 2yr 3yr P2 1.4 0.8 0.5 0.4 Cel 4.1 2.5 1.6 1.1 P3 40.3 24.5 15.9 10.9 P4 34.6 46.1 52.3 56.0 Itanium 3.9 5.2 5.9 6.3 AthlonMP 12.4 16.5 18.7 20.0 AthlonXP 1.3 1.8 2.0 2.1 Opteron 2.0 2.6 3.0 3.2 Percentage of Sample Set and Predicted Sample Set

24. 24 15 Resource Distribution: Memory Size, Cluster Nodes, SMPs Normal distribution

25. 25 15 Summary Models for resources Normal distribution SMPs, memory size, cluster hosts Processor architecture Need more study for the others Application VGrADS runtime system evaluation Description language Resource selection MicroGrid grid topology Network topology (Brite) + host configuration More Information [Kee&Chien, SC2004]

26. MicroGrid Update 9/13/200426