MicroGrid Update & A Synthetic Grid Resource Generator Xin Liu, Yang-suk Kee, Andrew Chien Department of Computer Science and Engineering Center for Networked.

Slides:

Advertisements

Similar presentations

-Grids and the OptIPuter Software Architecture Andrew A. Chien Director, Center for Networked Systems SAIC Chair Professor, Computer Science and Engineering.

Advertisements

A Workflow Engine with Multi-Level Parallelism Supports Qifeng Huang and Yan Huang School of Computer Science Cardiff University

Performance Testing - Kanwalpreet Singh.

1 The ns-2 Network Simulator H Plan: –Discuss discrete-event network simulation –Discuss ns-2 simulator in particular –Demonstration and examples: u Download,

Software-defined networking: Change is hard Ratul Mahajan with Chi-Yao Hong, Rohan Gandhi, Xin Jin, Harry Liu, Vijay Gill, Srikanth Kandula, Mohan Nanduri,

The Case for Enterprise Ready Virtual Private Clouds Timothy Wood, Alexandre Gerber *, K.K. Ramakrishnan *, Jacobus van der Merwe *, and Prashant Shenoy.

Esma Yildirim Department of Computer Engineering Fatih University Istanbul, Turkey DATACLOUD 2013.

September 21, Broadband Wireless Network Applications and Performance Carey Williamson Professor/iCORE Senior Research Fellow Department of Computer.

1 In VINI Veritas: Realistic and Controlled Network Experimentation Jennifer Rexford with Andy Bavier, Nick Feamster, Mark Huang, and Larry Peterson

15-441: Computer Networking Lecture 26: Networking Future.

Traffic Engineering With Traditional IP Routing Protocols

Shadow Configurations: A Network Management Primitive Richard Alimi, Ye Wang, Y. Richard Yang Laboratory of Networked Systems Yale University.

1 Internet Protocols and Network Performance Issues Carey Williamson iCORE Professor Department of Computer Science University of Calgary.

Shadow Configurations: A Network Management Primitive Richard Alimi, Ye Wang, and Y. Richard Yang Laboratory of Networked Systems Yale University February.

A Routing Control Platform for Managing IP Networks Jennifer Rexford Princeton University

The new The new MONARC Simulation Framework Iosif Legrand  California Institute of Technology.

1 Evgeny Bolotin – ICECS 2004 Automatic Hardware-Efficient SoC Integration by QoS Network on Chip Electrical Engineering Department, Technion, Haifa, Israel.

Network Monitoring for Internet Traffic Engineering Jennifer Rexford AT&T Labs – Research Florham Park, NJ 07932

Tesseract A 4D Network Control Plane

Dynamic Topology Adaptation of Virtual Networks of Virtual Machines Ananth I. Sundararaj Ashish Gupta Peter A. Dinda Prescience Lab Department of Computer.

Building a Strong Foundation for a Future Internet Jennifer Rexford ’91 Computer Science Department (and Electrical Engineering and the Center for IT Policy)

Testing Intrusion Detection Systems: A Critic for the 1998 and 1999 DARPA Intrusion Detection System Evaluations as Performed by Lincoln Laboratory By.

Edge Based Cloud Computing as a Feasible Network Paradigm(1/27) Edge-Based Cloud Computing as a Feasible Network Paradigm Joe Elizondo and Sam Palmer.

Active Network Applications Tom Anderson University of Washington.

Networking Virtualization Using FPGAs Russell Tessier, Deepak Unnikrishnan, Dong Yin, and Lixin Gao Reconfigurable Computing Group Department of Electrical.

Designing Efficient Systems Services and Primitives for Next-Generation Data-Centers K. Vaidyanathan, S. Narravula, P. Balaji and D. K. Panda Network Based.

Advanced Network Architecture Research Group 2001/11/149 th International Conference on Network Protocols Scalable Socket Buffer Tuning for High-Performance.

SCAN: a Scalable, Adaptive, Secure and Network-aware Content Distribution Network Yan Chen CS Department Northwestern University.

1 Enabling Large Scale Network Simulation with 100 Million Nodes using Grid Infrastructure Hiroyuki Ohsaki Graduate School of Information Sci. & Tech.

University of California, San Diego Computer Science and Engineering Concurrent Systems Architecture Group Agile Objects: Component-based Inherent Survivability.

The MicroGrid: A Scientific Tool for Modeling Grids Andrew A. Chien SAIC Chair Professor Department of Computer Science and Engineering University of California,

VeriFlow: Verifying Network-Wide Invariants in Real Time

The Center for Autonomic Computing is supported by the National Science Foundation under Grant No NSF CAC Seminannual Meeting, October 5 & 6,

Using Measurement Data to Construct a Network-Wide View Jennifer Rexford AT&T Labs—Research Florham Park, NJ

Resisting Denial-of-Service Attacks Using Overlay Networks Ju Wang Advisor: Andrew A. Chien Department of Computer Science and Engineering, University.

An Analysis of Location-Hiding Using Overlay Networks Ju Wang and Andrew A. Chien Department of Computer Science and Engineering, University of California.

Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed.

Management for IP-based Applications Mike Fisher BTexaCT Research

Simulation, Emulation Sathish Vadhiyar Sources / Credits: Microgrid, Simgrid.

Introduction to dCache Zhenping (Jane) Liu ATLAS Computing Facility, Physics Department Brookhaven National Lab 09/12 – 09/13, 2005 USATLAS Tier-1 & Tier-2.

Web Cache Redirection using a Layer-4 switch: Architecture, issues, tradeoffs, and trends Shirish Sathaye Vice-President of Engineering.

Advanced Network Architecture Research Group 2001/11/74 th Asia-Pacific Symposium on Information and Telecommunication Technologies Design and Implementation.

Wide-Area Service Composition: Performance, Availability and Scalability Bhaskaran Raman SAHARA, EECS, U.C.Berkeley Presentation at Ericsson, Jan 2002.

1 Evaluation of Cooperative Web Caching with Web Polygraph Ping Du and Jaspal Subhlok Department of Computer Science University of Houston presented at.

A Utility-based Approach to Scheduling Multimedia Streams in P2P Systems Fang Chen Computer Science Dept. University of California, Riverside

11 CLUSTERING AND AVAILABILITY Chapter 11. Chapter 11: CLUSTERING AND AVAILABILITY2 OVERVIEW  Describe the clustering capabilities of Microsoft Windows.

Plethora: Infrastructure and System Design. Introduction Peer-to-Peer (P2P) networks: –Self-organizing distributed systems –Nodes receive and provide.

1 Wide Area Network Emulation on the Millennium Bhaskaran Raman Yan Chen Weidong Cui Randy Katz {bhaskar, yanchen, wdc, Millennium.

A Grid-enabled Multi-server Network Game Architecture Tianqi Wang, Cho-Li Wang, Francis C.M.Lau Department of Computer Science and Information Systems.

An Efficient Gigabit Ethernet Switch Model for Large-Scale Simulation Dong (Kevin) Jin.

CS 6401 Overlay Networks Outline Overlay networks overview Routing overlays Resilient Overlay Networks Content Distribution Networks.

An Efficient Gigabit Ethernet Switch Model for Large-Scale Simulation Dong (Kevin) Jin.

By Miguel A. Erazo Advisor: Jason Liu March 2009.

1 PerfCenter and AutoPerf: Tools and Techniques for Modeling and Measurement of the Performance of Distributed Applications Varsha Apte Faculty Member,

On the Placement of Web Server Replicas Yu Cai. Paper On the Placement of Web Server Replicas Lili Qiu, Venkata N. Padmanabhan, Geoffrey M. Voelker Infocom.

Latest Improvements in the PROOF system Bleeding Edge Physics with Bleeding Edge Computing Fons Rademakers, Gerri Ganis, Jan Iwaszkiewicz CERN.

University of Texas at Arlington Scheduling and Load Balancing on the NASA Information Power Grid Sajal K. Das, Shailendra Kumar, Manish Arora Department.

TRUST Self-Organizing Systems Emin G ü n Sirer, Cornell University.

1 Traffic Engineering By Kavitha Ganapa. 2 Introduction Traffic engineering is concerned with the issue of performance evaluation and optimization of.

Internet Traffic Engineering Motivation: –The Fish problem, congested links. –Two properties of IP routing Destination based Local optimization TE: optimizing.

1 Evaluation of Cooperative Web Caching with Web Polygraph Ping Du and Jaspal Subhlok Department of Computer Science University of Houston presented at.

1 Scalability and Accuracy in a Large-Scale Network Emulator Nov. 12, 2003 Byung-Gon Chun.

VGES Demonstrations Andrew A. Chien, Henri Casanova, Yang-suk Kee, Richard Huang, Dionysis Logothetis, and Jerry Chou CSE, SDSC, and CNS University of.

Network Processing Systems Design

Accelerating Peer-to-Peer Networks for Video Streaming

Distributed Multimedia Systems

Monkey See, Monkey Do A Tool for TCP Tracing and Replaying

CLUSTER COMPUTING.

Cloud Web Filtering Platform

Performance And Scalability In Oracle9i And SQL Server 2000

Presentation transcript:

MicroGrid Update & A Synthetic Grid Resource Generator Xin Liu, Yang-suk Kee, Andrew Chien Department of Computer Science and Engineering Center for Networked Systems University of California, San Diego September 13-14, 2004 VGrADS Workshop University of Tennessee, Knoxville

MicroGrid Update 9/13/20042 Grid Application Virtual Grid, “MicroGrid” MicroGrid Software LAN Workgroup Scalable Cluster Heterogeneous Environment MicroGrid Enables Deep Study of Grid Dynamics

MicroGrid Update 9/13/20043 MicroGrid Highlights Binary Interception enables Transparent Virtualization (SC2000) »“Virtual time” enables wide range of relative performance experiments Scalable Packet-level Simulation provides accurate protocol behavior (SC2003) »Profile and Topology driven Graph Partitioners »Full TCP, Router, OSPF, and BGP modeling MicroGrid validated on diverse benchmarks & grid applications (JOGC 2004) Large-Scale ISP Simulation (20,000+ routers) based on new hierarchical load-balance and partition (SC2004) Source Releases: 2/2003, 7/2003, Version Feb 2004

MicroGrid Update 9/13/20044 Hierarchical Grouping and Load Balance For Further Scalability Scalability Challenges: »Non-linear effect of Minimal Link Latency –Lead to a small MLL »Poor Partition Results from METIS Solution: »Reduce the original graph by merging nodes with link latency less than a threshold The Key: »How to set the link latency threshold

MicroGrid Update 9/13/20045 Partition Efficiency (E) E= Es * Ec Es: efficiency decided by achieved MLL »Es = (MLL – Cn)/MLL »Cn: synchronization cost for n physical nodes Ec: efficiency decided by the load balance »Ec = Caverage/Cmax  Tradeoffs between Parallelism and Efficiency  Evaluate Different Partition Outputs w/o Running the Simulation

MicroGrid Update 9/13/20046 Scalable Simulation with Hierarchical Load Balance Hierarchical and Profile-based Partition/Balance enables high efficiency Scalable to 20,000 routers using 90 nodes on TeraGrid ~ 40% load balance, ~50% simulation time improvement ~45% parallel efficiency

MicroGrid Update 9/13/20047 High Fidelity Modeling Realistic Routing Structure »Flat networks of 20,000 routers (OSPF) »Hierarchical networks of 100 AS’s with 200 routers each (BGP & OSPF) BGP Configuration »Simulate Typical BGP Configuration Practice »Providers, Customers, Peering, Exporting »Realistic Structure »=> ongoing study of how real, open problem in networking community

MicroGrid Update 9/13/20048 Applications of MicroGrid Detailed Application Performance and Configuration Studies Resource Discovery and Selection Studies Scheduling Studies Adaptation (Rescheduling) Studies Desktop Grid (P2P Computing) Dynamic Resource Management and Trading Policies Application Deployment Studies Resource Configuration Studies … and Many More …

MicroGrid Update 9/13/20049 Challenging Example: Denial of Service User Edge Proxy Proxy Resource Pool (IP Network) Host Overlay Network Application Attacker

MicroGrid Update 9/13/ Challenges Large Traffics »Malicious attacks »Large overlay network Accuracy »Application latency/ throughput »Packet drop, Link congestion Flexibility »Various network topology »Effect of different overlay structure

MicroGrid Update 9/13/ Experiment Configuration Overlay Network » ForwardEngine Prototype Application »Apache server / Siege http clients Attackers »simTrinoo UDP traffic ~1000 routers, 64 node overlay network, 200 siege clients, 10GByte/s attacking traffic

MicroGrid Update 9/13/ Ongoing Experiments Application Performance »With and Without Attacking Traffic »Effect of Overlay Network Placement Resilience of DOS Attack »App Performance under Different Attack Magnitudes Client Failover Policy »How to achieve best performance under attack –Fix Binding –Random Selection –Best Achievable

MicroGrid Update 9/13/ Detailed Application Behavior All Details: Buffer filling, Packet Drop

MicroGrid Update 9/13/ Effect of Overlay Placement 2ms68ms3ms 22ms48ms 2ms48ms3ms22ms 67ms Baseline Case1 Case2 Case3 3ms App Server Proxy Nodes App Client

MicroGrid Update 9/13/ User Response Time Proxy can Reduce the Response Time! The Last Proxy should be Close to the App Server

MicroGrid Update 9/13/ User Bandwidth Proxy can also Increase the Throughput!

MicroGrid Update 9/13/ CDF of User Observed Response Time

MicroGrid Update 9/13/ Summary MicroGrid continues to improve in capability Integrated »Topology Generators (maBrite) »Background Traffic Generators »Automatic Profile-based Load Balance »BGP Configuration Scalable, Large-network Simulations are being achieved at detailed packet-level with Large-scale Parallel Resources

Synthetic Resource Generator for Computational Grids Yang-suk Kee and Andrew Chien Department of Computer Science and Engineering Center for Networked Systems University of California, San Diego [Kee&Chien, SC2004]

20 15 Needs of Synthetic Platform Generator Evaluation of Resource monitoring system Resource management technique Resource description language Resource selection(matching) algorithm Application scheduling algorithm How to get representative resource configuration scenarios? It is not feasible to build large reconfigurable Grids

21 15 Synthetic Platform Generator Is A tool to synthesize platform configurations for computation Grids by statistical analysis of distributions of existing grid resources Characteristics New models for Grid resource configuration 10,179 processors from 114 clusters Validation of the resource models 10,073 processors from 191 clusters Extrapolation to future Grid resource configuration

22 15 Resources of Interest Commodity-based clusters Processor architecture Processor clock speed Processor cache # of processors per node Memory size per node Disk capacity per node # of nodes per cluster System Area Network (SAN) per cluster

23 15 Resource Distribution: Processor Architecture Samples 1yr 2yr 3yr P Cel P P Itanium AthlonMP AthlonXP Opteron Percentage of Sample Set and Predicted Sample Set

24 15 Resource Distribution: Memory Size, Cluster Nodes, SMPs Normal distribution

25 15 Summary Models for resources Normal distribution SMPs, memory size, cluster hosts Processor architecture Need more study for the others Application VGrADS runtime system evaluation Description language Resource selection MicroGrid grid topology Network topology (Brite) + host configuration More Information [Kee&Chien, SC2004]

MicroGrid Update 9/13/200426