Grid Economics and Business Models: A Gridbus Perspective Rajkumar Buyya Grid and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne Melbourne, Australia WW Grid Grid User Meet, Bengaluru, India

2 Outline Introduction Utility Networks and Grid Computing Global Grids and Challenges Security. Resource management, pricing and service models Service Oriented Grids and Grid Economy SOGA, Grid Market Directory, Grid Bank, Broker Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary

3 4 Essential Utilities and Delivery Networks (1) Water (2) Electricity (3) Gas (4) Telephone

4 (5) IT services as the fifth utility (water, electricity, gas, telephone, IT) eScience eBusiness eGovernment eHealth Multilingual eEducation …

5 Leading to Grid (computing) Paradigm: Cyberinfrastructure for sharing resources Inspired by Power Grid! * A service-oriented/utility computing paradigm that enables seamless sharing of geographically distributed, autonomous resources (for PROFIT). * This was the original aim of building Internet although it ended up in giving birth to !

6 A Bird Eye View of World-Wide Grid Environment Grid Resource Broker Resource Broker Application Grid Information Service Grid Resource Broker database R2R2 R3R3 RNRN R1R1 R4R4 R5R5 R6R6 Grid Information Service

7 Various Types of Grid Services Computational Services – CPU cycles NASA IPG, TeraGrid, I-Grid, … Data Services Data replication, management, secure access-- LHC Grid/Napster Application Services Access to remote software/libraries and license management NetSolve Interaction Services eLearning, Virtual Tables, Group Communication (Access Grid), Gaming Knowledge Services The way knowledge is acquired and managed data mining. Utility Computing Services Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities. Computational Grid Data Grid ASP Grid Interaction Grid Knowledge Grid Utility Grid

8 Worldwide Grid Spending After the year 2006, business popularity of Grid computing is expected to be accelerate exponentially: Especially, the financial services and ERP services is expected to take major parts in the expense Source: Insight Research Corp.

9 Grid Challenges Security Resource Allocation & Scheduling Data locality Network Management System Management Resource Discovery Uniform Access Computational Economy Application Construction

10 Some Grid Initiatives Worldwide Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch Brazil OurGrid, EasyGrid LNCC-Grid + many others China ChinaGrid – Education CNGrid - application Europe UK eScience EU Grids.. and many more... India Garuda Japan NAGERI Korea... N*Grid Singapore NGP USA Globus NASA IPG AccessGrid TeraGrid Cyberinfrasture Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft -.NET Oracle 10g Satyam – Grid Practice Infosys, Wipro, TCS StorageTek – Grid.. Public Forums Global Grid Forum Australian Grid Forum Conferences: CCGrid Grid HPDC E-Science billion – 3 yrs 1 billion – 5 yrs 450million – 5 yrs 486million – 5 yrs 1.3 billion (Rs) 27 million 2? billion 120million – 5 yrs

11 Grid (Market) Participant Types and Application Category commercial scientific free trading regulation Public computing Alchemi, UD) National provider (K*Grid, TeraGrid, Garuda/IndiaGrid, UKGrid, AusGrid) Private enterprises (Satyam, IBM, Sun) Application Category Sharing Model

12 mix-and-match (service) Object-oriented Internet/partial-P2P Network enabled Solvers Economic-based Utility / Service-Oriented Computing Nimrod-G

13 The Gridbus Melbourne: Enable Leasing of ICT Services on Demand WWG World Wide Grid! On Demand Utility Computing Gridbus Distributed Data

14

15 Grid Economy: Methodology for Sustained Resourced Sharing and Managing Supply-and-Demand for Resources

16 New challenges of Grid Economy Resource Owners How do I decide prices ? (economic models?) How do I specify them ? How do I translate price to resource allocation ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? ….. Resource Consumers How do I decide expenses ? How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I map jobs to resources to meet my QoS needs? ….. They need mechanisms and technologies for value expression, value translation, and value enforcement.

17 Grid Entities and Architecture GSP site scheduler accounting Grid consumer Market Maker GSP global scheduler broker Resource Provider End users Private enterprises National providers GSP site scheduler Resource Provider

18 Grid Node N A Reference Service-Oriented Architecture for Utility Grids Grid Consumer Programming Environments Grid Resource Broker Grid Service Providers Grid Explorer Schedule Advisor Trade Manager Job Control Agent Deployment Agent Trade Server Resource Allocation Resource Reservation R1R1 Misc. services Information Service R2R2 RmRm … Pricing Algorithms Accounting Grid Node1 … Grid Middleware Services … … Health Monitor Grid Market Services JobExec Info ? Secure Trading QoS Storage Sign-on Grid Bank Applications Data Catalogue

19 Gridbus and Complementary Technologies – realizing Utility Grid AIX Solaris WindowsLinux.NET Grid Fabric Software Grid Applications Core Grid Middleware User-Level Middleware (Grid Tools) Grid Bank Grid Exchange & Federation JVM Grid Brokers: X-Parameter Sweep Lang. Gridbus Data Broker MPI CondorSGETomcatPBS Alchemi Workflow IRIXOSF1 Mac Libra GlobusUnicore … … Grid Market Directory PDBCDB Worldwide Grid Grid Fabric Hardware … … PortalsScienceCommerceEngineering … … Collaboratories … … Workflow Engine Grid Storage Economy Grid Economy NorduGridXGrid ExcellGrid Nimrod-G GRIDSIMGRIDSIM Gridscape

20 On Demand Assembly of Services: Putting Them All Together ASP Catalogue Grid Info Service Grid Market Directory GSP (Accounting Service) Gridbus GridBank GSP (e.g., UofM) PE GSP (e.g., VPAC) PE GSP (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Alchemi GS GTS Cluster Scheduler Job 8 Grid Resource Broker 2 Visual Application Composer Application Code Explore data Results 97 Results+ Cost Info Bill 12 Data Catalogue

21 Alchemi:.NET-based Enterprise Grid Platform & Web Services Internet Alchemi Worker Agents Alchemi Manager Alchemi Users Web Services like Model General Purpose Dedicated/Non-dedicate workers Role-based Security.NET and Web Services C# Implementation GridThread and Job Model Programming Easy to setup and use Widely in use!

22 Some Users of Alchemi Tier TechnologiesTier Technologies, USA Large scale document processing using Alchemi framework CSIROCSIRO, Australia Natural Resource Modeling The Friedrich Miescher Institute (FMI) for Biomedical ResearchThe Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland Patterns of transcription factors in mammalian genes Satyam Computers Applied Research LaboratorySatyam Computers Applied Research Laboratory, India Micro-array data processing using Alchemi framework The University of Sao PauloThe University of Sao Paulo, Brazil The Alchemi Executor as a Windows Service stochastix GmbHstochastix GmbH, Germany Asynchronous Excel Tasks using ManagedXLL and Alchemi.Net Grid Computing framework. Many users in Universities: See next for an example.

The Gridbus Grid Service Broker for Data Grid Applications Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, ] And Extends its notion for Data and Service Grids

24 Gridbus Broker Architecture Grid Middleware Gridbus Client Gribus Client Grid Info Server Schedule Advisor Trading Manager Gridbus Farming Engine Record Keeper Grid Explorer GE GIS, NWS TM TS RM & TS Grid Dispatcher RM: Local Resource Manager, TS: Trade Server G G C U Globus enabled node. A L Alchemi enabled node. (Data Grid Scheduler) Data Catalog Data Node Unicore enabled node. $ $ $ App, T, $, Opt (Bag of Tasks Applications)

25 Gridbus Broker and Remote Service Access Enablers Alchemi Gateway UnicoreData Store Access Technology Grid FTP SRB -PBS -Condor -SGE Globus Job manager fork()batch() Gridbus agent Data Catalog -PBS -Condor -SGE -XGrid SSH fork() batch() Gridbus agent Credential Repository MyProxy Home Node/Portal Gridbus Broker fork() batch() -PBS -Condor -SGE -Alchemi -XGrid Portlets

26 Gridbus Services for eScience applications Application Development Environment: XML-based language for composition of task farming (legacy) applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine. Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes that meet user QoS requirements. Hide L ow-Level Grid Middleware interfaces Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.

27 Drug Design Made Easy! Click Here for Demo

Economy-based Data Grid Scheduling High Energy Physics as eScience Application Case Study CLICK HERE TO SKIP IF RUNNING OUT of TIME

29 Case Study: High Energy Physics and Data Grid The Belle Experiment KEK B-Factory, Japan Investigating fundamental violation of symmetry in nature (Charge Parity) which may help explain the universal matter – antimatter imbalance. Collaboration 400 people, 50 institutes 100 s TB data currently

30 Australian Belle Data Grid Testbed VPAC Melbourne

31 Case Study: Event Simulation and Analysis B0->D*+D*-Ks Simulation and Analysis Package - Belle Analysis Software Framework (BASF) Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data Analyzed 100 data files (30MB each) were distributed among the five nodes

32 Belle Data Grid (GSP CPU Service Price: G$/sec) NA G$4 Data node G$6 VPAC Melbourne G$2

33 Belle Data Grid (Bandwidth Price: G$/MB) NA G$4 Data node G$6 VPAC Melbourne G$

34 Network Cost (in Grid $/Currency!)

35 Deploying Application Scenario A data grid scenario with 100 jobs and each accessing remote data of ~30MB Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario: Minimise Time Minimise Cost Results:

36 Time Minimization in Data Grids Time (in mins.) Number of jobs completed fleagle.ph.unimelb.edu.aubelle.anu.edu.aubelle.physics.usyd.edu.aubrecca-2.vpac.org

37 Results : Cost Minimization in Data Grids Time(in mins.) Number of jobs completed fleagle.ph.unimelb.edu.aubelle.anu.edu.aubelle.physics.usyd.edu.aubrecca-2.vpac.org

38 Observation Organization Node detailsCost (in G$/CPU- sec) Total Jobs Executed TimeCost CS,UniMelbbelle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource) -- Physics, UniMelbfleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, Linux CS, University of Adelaide belle.cs.adelaide.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource) -- ANU, Canberrabelle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, Linux 42 2 Dept of Physics, USyd belle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux VPAC, Melbourne brecca-2.vpac.org 180 node cluster (only head node used), Linux 623 2

39 Grid and Gridbus Technologies for Various Grid (Market) Types commercial scientific free trading regulation Public computin g (Alchemi) National provider (Globus, Gridbus,..) Private enterprises (Libra, Gridbus, Globus) Application Category Sharing Model

40 (5) IT services as the fifth utility (water, electricity, gas, telephone, IT) eScience eBusiness eGovernment eHealth Multilingual eEducation …

41 Summary and Conclusion Grids exploit synergies that result from cooperation of autonomous entities: Resource sharing, dynamic provisioning, and aggregation at global level. Grid Economy provides incentive needed for sustained cooperation. Grid Network has potential to serve as Cyberinfrastructure for Utility Computing Grids offer enormous opportunities for realizing eScience and eBusiness at global level.

42 Any Questions ? Gridbus Project -

43 Thanks for your attention! The Gridbus Cooperation!

Backup Slides

45 This talk is designed to answer: How can Grid technologies support the emergence and operation of virtual enterprises? How can Grid shared resources be treated, brokered, and marketed as ICT commodities or futures among networked organisations? What kind of Grid architecture is needed for handling such market mechanisms in an automated fashion? How can Grid economies map the evolution of networked business models?

46 Realising Market-based Grid: Minimal New Components Grid Market Directory Services Grid Trading Services – for different economic models Grid Metering Services Grid Accounting and Payment Services Grid Service Broker

47 Deadline (D) and Budget (B) Constrained Scheduling Algorithms AlgorithmExecution Time (D) Execution Cost (B) Compute Grid Data Grid Cost OptLimited by DMinimize Yes Cost-Time Opt Minimize if possible Minimize Yes Time OptMinimizeLimited by B Yes Conservative -Time Opt MinimizeLimited by B, jobs have guaranteed minimum budget Yes