Convergence Characteristics for Clusters, Grids, and P2P networks Panel at the P2P conference, Linkoping, Sweden WW Grid Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab. The University of Melbourne Melbourne, Australia www.gridbus.org

Where Did I come From ? Stockholm+Linkoping * London * Singapore * * Melbourne

GRIDS Lab @ the U. of Melbourne, The Gridbus Project: www.gridbus.org Conduct R&D in Distributed Computational Economy for Service-Oriented Computing: Architecture, Specification, and Open Source Reference Implementation in in Collaboration with Any Global Peers. Platform: Cluster, Grid, P2P for diverse applications. Distributed Application Composition Tools Grid Economy & Scheduling (via Nimrod-G Broker) GridSim Toolkit: Grid Modeling and Simulation (Java based): Libra: Economic Cluster Scheduler Grid Bank: Accounting and Transaction Management World Wide Grid (WWG) – A Global Testbed P2P/Jxta-based Compute Power Market Application Enabler Projects: Virtual Laboratory Toolset for Drug Design High-Energy Physics and the Grid Network (HEPGrid) Brain Activity Analysis on the Grid Future: GridEmail for Spam Management —read only if sender pays!

Books

Technology Evolution: Cluster, Grid, P2P

Scalable Seamless Computing: Breaking Administrative Barriers 2100 ? PERFORMANCE 2100 Administrative Barriers Individual Group Department Campus State National Globe Inter Planet Galaxy Desktop SMPs or SuperComputers Local Cluster Enterprise Cluster/Grid Global Cluster/Grid Inter Planetary Grid!

Basic Elements Application Development Tools Security Uniform Access Security System Management Computational Economy Resource Discovery Resource Allocation & Scheduling Data locality Network Management Application Development Tools

Cluster Architecture Parallel Applications Parallel Applications Sequential Applications Sequential Applications Sequential Applications Parallel Programming Environment Cluster Middleware (Single System Image and Availability Infrastructure) PC/Workstation Network Interface Hardware Communications Software PC/Workstation Network Interface Hardware Communications Software PC/Workstation Network Interface Hardware Communications Software PC/Workstation Network Interface Hardware Communications Software Cluster Interconnection Network/Switch

Cluster Architecture Parallel Applications Parallel Applications Sequential Applications Sequential Applications Sequential Applications Parallel Programming Environment Cluster Middleware (Single System Image and Availability Infrastructure) PC/Workstation Network Interface Hardware Communications Software PC/Workstation Network Interface Hardware Communications Software PC/Workstation Network Interface Hardware Communications Software PC/Workstation Network Interface Hardware Communications Software Cluster Interconnection Network/Switch

A Typical Grid Computing Environment Grid Information Service Grid Resource Broker database Application R2 2 R3 R4 R5 RN Grid Resource Broker R6 R1 Resource Broker Grid Information Service

A Typical Grid Computing Environment Grid Information Service Grid Resource Broker database Application R2 2 R3 R4 R5 RN Grid Resource Broker R6 R1 Resource Broker Grid Information Service

A Typical P2P Computing Environment Peer Discovery Service Peer Agent Application P3 pM Who can help ? Peer P2, P7 can help! pN Request P2 Sorry, I am busy. Peer Agent Request Peer Agent Response P1 R7 p4 p5

A Typical P2P Computing Environment Peer Discovery Service Peer Agent Application P3 pM Who can help ? Peer P2, P7 can help! pN Request P2 Sorry, I am busy. Peer Agent Request Peer Agent Response P1 R7 p4 p5

Cluster, Grid, P2P: Characteristics Population Commodity Computers High-end computers Edge of network (desktop PC) Ownership Single Multiple Discovery Membership Services Centralised Index & Decentralised Info Decentralized User Management Centralised Decentralised Resource management Centralized Distributed Allocation/Scheduling Inter-Operability VIA based? No standards yet No standards Single System Image Yes No Scalability 100s 1000? Millions? [@Home] Capacity Guaranteed Varies, but high Varies Throughput Medium High Very High Speed(Lat. Bandwidth) Low, high High, Low

Nimrod-G: Modeling Distributed Computing in Economic Terms Nimrod/G Client Nimrod/G Client Nimrod/G Client Nimrod-G Engine Schedule Advisor Grid Store Trading Manager Grid Dispatcher Grid Explorer Grid Middleware Globus, Legion, Condor, etc. TM TS GE GIS Grid Information Server(s) RM & TS RM & TS RM & TS G C L G Legion enabled node. Globus enabled node. L G C L RM: Local Resource Manager, TS: Trade Server Condor enabled node.

On-Demand Computing Cost Deadline Legion hosts Globus Hosts Bezek is in both Globus and Legion Domains

Active Sheet  Nimrod-G: Transparent Spreadsheet Processing from PCs to Clusters to World-Wide Grids Proxy Nimrod-G World-Wide Grid

[Collaboration with Osaka University, Japan] MEG(MagnetoEncephaloGraphy) Data Analysis on the Grid: Brain Activity Analysis 64 sensors MEG Analysis All pairs (64x64) of MEG data by shifting the temporal region of MEG data over time: 0 to 29750: 64x64x29750 jobs 2 Data Generation 3 1 Data Analysis 5 Results Nimrod-G 4 [deadline, budget, optimization preference] Life-electronics laboratory, AIST World-Wide Grid Provision of expertise in the analysis of brain function Provision of MEG analysis [Collaboration with Osaka University, Japan]

Backup Slides

Evolution Time Line 1960 1970 1975 1980 1985 1990 1995 2000 * Sputnik * ARPANET * Email * Ethernet * TCP/IP * IETF * Internet Era * WWW Era * Mosaic * XML * PC Clusters Crays MPPs Mainframes * HTML * W3C P2P Grids XEROX PARC worm COMPUTING NETWORKING * Web Services Minicomputers PCs WS Clusters PDAs Workstations HTC

What is Cluster/Grid ? A type of parallel and distributed system that enables the sharing, selection, & aggregation of resources distributed in _____/_______ administrative domains depending on their availability, capability, performance, cost, and users quality of service requirements. Multiple A Single Cluster Grid

CPM: DC Economy-based P2P Computing (Jxta based Implementation) Market Server Market Repository Discovery - Membership CPM Agent User (Consumer) Bill Trader Job Management Resources (Provider) Accounting

Drug Design: Data Intensive Computing on Grid Protein Molecules Chemical Databases (legacy, in .MOL2 format) It involves screening millions of chemical compounds (molecules) in the Chemical DataBase (CDB) to identify those having potential to serve as drug candidates. [Collaboration with WEHI for Medical Science, Melbourne]

Virtual Drug Design A Virtual Lab for “Molecular Modeling for Drug Design” on P2P Grid Data Replica Catalogue Grid Market Directory Grid Info. Service “Give me list PDBs sources Of type aldrich_300?” “service cost?” “service providers?” GTS Resource Broker “Screen 2K molecules in 30min. for $10” “mol.5 please?” GTS (RB maps suitable Grid nodes and Protein DataBank) “get mol.10 from pdb1 & screen it.” PDB2 GTS “mol.10 please?” GTS GTS (GTS - Grid Trade Server) PDB1