1 Another Perspective on Grid Problem and Challenges ( A view from the Globus Project: ) M.Effatparvar Fall 1391

2 The Grid “ Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations” 1. Enable integration of distributed resources 2. Using general-purpose protocols & infrastructure 3. To achieve better-than-best-effort service

3 The Grid (2) Dynamically link resources/services From collaborators, customers, eUtilities, … (members of evolving “virtual organization”) Into a “virtual computing system” Dynamic, multi-faceted system spanning institutions and industries Configured to meet instantaneous needs, for: Multi-faceted QoS for demanding workloads Security, performance, reliability, …

4 Resource/Service Integration as a Fundamental Challenge R Discovery Many sources of data, services, computation R Registries organize services of interest to a community Access Data integration activities may require access to, & exploration/analysis of, data at many locations Exploration & analysis may involve complex, multi-step workflows RM Resource management is needed to ensure progress & arbitrate competing demands Security service Security service Policy service Policy service Security & policy must underlie access & management decisions

5 Some Grid Software Technologies

6 Grid Middleware Technologies Globus – Argonne National Lab and ISI Gridbus – University of Melbourne Unicore – Germany Legion – University of Virginia

7 Globus Toolkit

8 Globus Toolkit Services Grid Security Infrastructure (GSI) PKI-based Security (Authentication) Service Globus Resource Allocation Manager (GRAM) Uniform Job Submission Metacomputing Directory Service (MDS) LDAP-based Information Service Global Access to Secondary Storage (GASS) Remote Storage Access Service Remote Data Catalogue and Management Tools Support in Globus 2.0 (released in 2002)

9 GRAM Components Globus Security Infrastructure Job Manager GRAM client API calls to request resource allocation and process creation. MDS client API calls to locate resources Query current status of resource Create RSL Library Parse Request Allocate & create processes Process Monitor & control Site boundary ClientMDS: Grid Index Info Server Gatekeeper MDS: Grid Resource Info Server Local Resource Manager MDS client API calls to get resource info GRAM client API state change callbacks

10 Sample of High-Level Services Resource brokers and co-allocators DUROC, Nimrod-G, Gridbus, Condor-G, AppLeS PST Communication & I/O libraries MPICH-G, RIO (MPI-IO) Parallel languages HPC++, CC++ Collaborative environments CAVERNsoft, ManyWorlds Others MetaNEOS, NetSolve, LSA, AutoPilot, WebFlow

11 The Gridbus Melbourne: Enable Leasing of ICT Services on Demand WWG World Wide Grid!  On Demand Utility Computing Gridbus Distributed Data

12 Gridbus Architecture Layer

13 Gridbus and Complementary Grid Technologies AIX Solaris WindowsLinux.NET Grid Fabric Software Grid Applications Core Grid Middleware User-Level Middleware (Grid Tools) Grid Bank Grid Exchange & Federation JVM Grid Brokers: X-Parameter Sweep Lang. Gridbus Data Broker MPI CondorSGETomcatPBS Alchemi Workflow IRIXOSF1 Mac Libra GlobusUnicore … … Grid Market Directory PDBCDB Worldwide Grid Grid Fabric Hardware … … PortalsScienceCommerceEngineering … … Collaboratories … … Workflow Engine Grid Storage Economy Grid Economy NorduGridXGrid ExcellGrid Nimrod-G GRIDSIMGRIDSIM Gridscape

14 Putting them All Together: On Demand Assembly of Services Data Source (Instruments /distributed sources) Data Replicator (GDMP) ASP Catalogue Grid Info Service Grid Market Directory GSP (Accounting Service) Gridbus GridBank Data GSP (e.g., UofM) PE GSP (e.g., VPAC) PE GSP (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Alchemi GS GTS Cluster Scheduler Grid Service Provider (GSP) (e.g., CERN) PE Cluster Scheduler Job 8 Grid Resource Broker 2 Visual Application Composer Application Code Explore data Results 97 Results+ Cost Info Bill 12 Data Catalogue

15 Comparison of Middleware Technologies Middleware Property UNICOREGlobusLegionGridbus FocusHigh level Programming models Low level servicesHigh level Programming modelsAbstractions and market models CategoryMainly uniform job submission and monitoring Generic computational ArchitectureVertical multi tiered system Layered and modular toolkitVertically integrated systemLayered component and utility model Implementation Model Abstract Job ObjectHourglass model at system levelObject-oriented metasystemHourglass model at user level Implementation Technologies JavaC and JavaC++C, Java, C# and Perl Runtime PlatformUnix Unix and Windows with.NET (Alchemi) Programming Environment Workflow environmentReplacement libraries for Unix & C libraries. Special MPI library (MPICH – G), CoG (Commodity Grid) kits in Java, Python, CORBA, Matlab, Java Server Pages, Perl and Web Services Legion Application Programming Interfaces (API). Command line utilities Broker Java API XML-based parameter-sweep language Grid Thread model via Alchemi. Distribution Model Open source Not open source. Commercial version available Open source Some Users and Applications EuroGrid [18], Grid Interoperability Project (GRIP) [20], OpenMolGrid [19], and Japanese NAREGI [22]. AppLeS [28], Ninf [30], Nimrod-G [29], NASA IPG [36], Condor-G [31], Gridbus Broker [32], UK eScience Project [33], GriPhyN [35], and EU Data Grid [34]. NPACI Testbed [42], Nimrod-L [41], and NCBioGrid [40]. Additionally, it has been used in the study of axially symmetric steady flow [39] and protein folding [38] applications. ePhysics Portal [52], Belle Analysis Data Grid[50], NeuroGrid [48], Natural Language Engineering [53], HydroGrid [46], and Amsterdam Private Grid [47].

16 Grid Applications: Composition and Deployment – A Broker Perspective Nimrod-G Broker: A Grid Broker for Computational Grids Gridbus Broker: A Grid Service Broker for Data Grids

17 Grid Applications and Parametric Computing Bioinformatics: Drug Design / Protein Modelling Bioinformatics: Drug Design / Protein Modelling Sensitivity experiments on smog formation Natural Language Engineering Ecological Modelling: Control Strategies for Cattle Tick Electronic CAD: Field Programmable Gate Arrays Computer Graphics: Ray Tracing High Energy Physics: Searching for Rare Events Finance: Investment Risk Analysis VLSI Design: SPICE Simulations Aerospace: Wing Design Network Simulation Automobile: Crash Simulation Data Mining Civil Engineering: Building Design astrophysics

18 The Gridbus Grid Service Broker for Data Grid Applications Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, ] And Extends its notion for Data and Service Grids

19 A resource broker for scheduling task farming Data Grid applications with static or dynamic parameter sweeps on global Grids. It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation) Key Features A single window to manage & control experiment Programmable Task Farming Engine Resource Discovery and Resource Trading Optimal Data Source Discovery Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & sharing of results Accounting Grid Service Broker (GSB)

20 Gridbus Broker Architecture Grid Middleware Gridbus Client Gribus Client Grid Info Server Schedule Advisor Trading Manager Gridbus Farming Engine Record Keeper Grid Explorer GE GIS, NWS TM TS RM & TS Grid Dispatcher RM: Local Resource Manager, TS: Trade Server G G C U Globus enabled node. A L Alchemi enabled node. (Data Grid Scheduler) Data Catalog Data Node Unicore enabled node. $ $ $ App, T, $, Opt (Bag of Tasks Applications)

21 Portal-based Access to Grid Broker for Launching and Steering Applications Grid Broker World-Wide Grid

22 Excel Plugin to Access Gridbus Services Excel ExcelGrid Add-In ExcelGrid Runner ExcelGridJob ExcelGrid MiddlewareGridbus BrokerEnterprise Grid 210 0

23 The Grid Impact! “The global computational grid is expected to drive the economy of the 21 st century similar to the electric power grid that drove the economy of the 20 th century”

24 References Mark Baker, Rajkumar Buyya, and Domenico Laforenza, Grids and Grid Technologies for Wide-Area Distributed Computing, International Journal of Software: Practice and Experience (SPE), Volume 32, Issue 15, Wiley Press, USA, Nov Grids and Grid Technologies for Wide-Area Distributed Computing Madhu Chetty and Rajkumar Buyya, Weaving Computational Grids: How Analogous Are They with Electrical Grids?, Computing in Science and Engineering (CiSE), ISSN , Volume 4, Issue 4, Pages: 61-71, IEEE CS Press and American Institute of Physics, USA, July-August 2002.Weaving Computational Grids: How Analogous Are They with Electrical Grids?CiSE Ian Foster, Carl Kesselman, Steve Tuecke, The Anatomy of the Grid: Enabling Scalable Virtual Organizations, International J. Supercomputer Applications, 15(3), 2001.The Anatomy of the Grid: Enabling Scalable Virtual Organizations Parvin Asadzadeh, Rajkumar Buyya, Chun Ling Kei, Deepa Nayar, and Srikumar Venugopal, Global Grids and Software Toolkits: A Study of Four Grid Middleware Technologies, High Performance Computing: Paradigm and Infrastructure, Laurence Yang and Minyi Guo (editors), ISBN: X, Wiley Press, New Jersey, USA, June 2005.Global Grids and Software Toolkits: A Study of Four Grid Middleware Technologies