Grid Computing Research @ Melbourne: Gridbus Perspective Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.gridbus.org Gridbus Sponsors

Grids @ Various Schools in Melbourne: Technology Creation -> Applications Faculty of Engineering – CS/EE/Geomatics GRIDS Lab and the Gridbus Project – technology focus SensorWeb (NICTA) Natural Language Processing Virtual organisation for tsunami research Faculty of Science - Physics Belle and LHC Grid Australian Virtual Observatory Faculty of Medicine – HFI, Anatomy MRI Grid Distributed Kidney Modelling Social Sciences/Humanities The development of a collaborative distributed environment for humanities research projects whose primary data takes the form of ethnographic audiovisual media. A Grid-enabled technology for the sharing of graphics, audio, video and text between social science research groups working internationally. Information Tech. Services Division MARC (Melbourne Advanced Research Computing)/HPC

Classes of Grid Services Computational Services – CPU cycles Pooling computing power: SETI@Home, TeraGrid, AusGrid, ChinaGrid, IndiaGrid, UK Grid,… Data Services Collaborative data sharing generated by instruments, sensors, persons: LHC Grid, Napster Application Services Access to remote software/libraries and license management—NetSolve Interaction Services eLearning, Virtual Tables, Group Communication (Access Grid), Gaming Knowledge Services The way knowledge is acquired, processed and managed—data mining. Utility Computing Services Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities. Utility Grid Users Knowledge Grid Interaction Grid ASP Grid Data Grid infrastructure Computational Grid

Grid Challenges Application Construction Security Uniform Access Security System Management Computational Economy Resource Discovery Resource Allocation & Scheduling Data locality Network Management Application Construction

+ Community Services: e.g., IEEE TC for Scalable Computing GRIDS Lab @ Melbourne R & D Education Youngest and one of the rapidly growing research labs in our School/University: Founded in 2002 Houses: Research Fellows/PostDocs Research Programmers PhD candidates Honours/Masters students Funding National and International organizations Australian Research Council & DEST Many industries (Sun, StorageTek, Microsoft, IBM, Microsoft) University-wide collaboration: Faculties of Science, Engineering, and Medicine Many national and international collaborations. Academics Industries Software: Widely in academic and industrial users. Publication: GRIDS Lab produces over 20% of our Dept’s research output. + Community Services: e.g., IEEE TC for Scalable Computing

Selected Funded Projects: 2004-2006 Australian Research Council: Discovery Projects: "The Development of Data Grids for High Energy Physics" "Economic Scheduling for Efficient Management of Clusters and their Cooperative Federation" Application Projects: "Development of e-Research Tools for an MRI Grid Computing Facility" "e-Research Grid Environment for Integration of Distributed Kidney Models and Resources”

Pushes Grid computing into mainstream computing The Gridbus Project @ Melbourne: Enable Leasing of ICT Services on Demand WWG Gridbus Pushes Grid computing into mainstream computing

What does Grid players require? They need tools and technologies that help them in value expression, value translation, and value enforcement. Grid Service Consumers (GSCs): How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I map jobs to resources to meet my QoS needs? How do I manage Grid dynamics and get my work done? … Grid Service Providers (GSPs) How do I decide service pricing models ? How do I specify them ? How do I translate them into resource allocations ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments?

Solution 1: Service Oriented Architecture (SOA) A SOA is a contractual architecture for offering and consuming software as services. There are four entities that make up an SOA service provider, service registry, and service consumer (also known as service requestor). The functions or tasks that the service provider offers, along with other functional and technical information required for consumption, are defined in the service definition or contract. registry contract provider consumer

Solution 2: Market-Oriented Grid Computing - (a) Sustained Resourced Sharing and (b) Effective Management of Shared Resources For resource providers, they provide service with different goals For consumers, they have different requirments How to manage grid service supply-and-demand is hard task So how can consumers find a resource depending on their requirements. And what type of resource that providers provide and how can they serve consumers within certain QoS. For example, when many consumers want to access same resource, who should be served first How to manage supply-demand balance is another challenge. Grid Economy

Service-Oriented Grid Architecture Data Catalogue Grid Bank Grid Market Services Information Service Sign-on Health Monitor Info ? Grid Explorer Grid Node N … … Programming Environments Secure Job Control Agent Applications Schedule Advisor Grid Node1 QoS Pricing Algorithms Trade Server Trading Trade Manager Accounting Resource Reservation Misc. services … Deployment Agent JobExec Resource Allocation Grid Resource Broker Storage R1 R2 … Rm Core Middleware Services Grid Service Consumer Grid Service Providers

Gridbus and Complementary Technologies – realizing Utility Grid … Grid Applications Science Commerce Engineering Collaboratories Portals … MPI ExcellGrid Gridscape Workflow X-Parameter Sweep Lang. User-Level Middleware … Grid Brokers: Nimrod-G Workflow Engine Gridbus Data Broker Globus Unicore Grid Storage Economy Grid Exchange & Federation Grid Market Directory Core Grid Middleware … Grid Bank Alchemi NorduGrid XGrid Grid Economy .NET JVM Condor PBS SGE Libra Tomcat Grid Fabric Software Windows Solaris Linux AIX IRIX OSF1 Mac CDB Grid Fabric Hardware PDB Worldwide Grid

On Demand Assembly of Services: Putting Them All Together Visual Application Composer Application Code Explore data 1 Results+ Cost Info 10 Grid Resource Broker 2 Bill 12 Data Catalogue 5 ASP Catalogue Grid Info Service Grid Market Directory 4 6 3 Job 8 Results 9 7 GSP (e.g., UofM) PE (e.g., VPAC) (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Alchemi GS GTS Cluster Scheduler GSP (Accounting Service) Gridbus GridBank 11

Alchemi: .NET-based Enterprise Grid Platform & Web Services Alchemi Manager Web Services Internet Alchemi Users Internet SETI@Home like Model General Purpose Dedicated/Non-dedicate workers Role-based Security .NET and Web Services C# Implementation GridThread and Job Model Programming Easy to setup and use Widely in use! Alchemi Worker Agents

Many users in Universities: See next for an example. Some Users of Alchemi Tier Technologies, USA Large scale document processing using Alchemi framework Satyam Computers Applied Research Laboratory, India Micro-array data processing using Alchemi framework CSIRO, Australia Natural Resource Modeling The University of Sao Paulo, Brazil The Alchemi Executor as a Windows Service stochastix GmbH, Germany Serving clients in International Banking/Finance sector The Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland Patterns of transcription factors in mammalian genes Many users in Universities: See next for an example.

Agenda Introduction Global Grids and Challenges Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services Global Grids and Challenges Security, resource management, pricing models, … Service-Oriented Grid Architecture and Gridbus Solutions Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker Architecture, Design and Implementation Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids A Case Study in High Energy Physics Summary and Conclusion

Grid Service Broker (GSB) A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids. It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation) Key Features A single window to manage & control experiment Programmable Task Farming Engine Resource Discovery and Resource Trading Optimal Data Source Discovery Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & sharing of results Accounting

Gridbus Broker Architecture Gridbus Client Gribus Client Gridbus Client (Bag of Tasks Applications) App, T, $, Opt (Data Grid Scheduler) Gridbus Farming Engine Schedule Advisor Record Keeper Trading Manager Grid Dispatcher Grid Explorer Grid Middleware TM TS $ GE GIS, NWS Grid Info Server RM & TS G $ Data Catalog Data Node C $ U G Unicore enabled node. Globus enabled node. L A RM: Local Resource Manager, TS: Trade Server Alchemi enabled node.

Gridbus Broker: Separating “applications” from “different” remote service access enablers and schedulers Application Development Interface Home Node/Portal Gridbus Broker fork() batch() -PBS -Condor -SGE -Alchemi -XGrid Single-sign on security Alogorithm1 Scheduling Interfaces AlogorithmN Data Catalog Plugin Actuators Alchemi -PBS -Condor -SGE Globus Job manager fork() batch() Gridbus agent Data Store Gateway Unicore -PBS -Condor -SGE -XGrid SSH fork() batch() Gridbus agent Access Technology SRB Grid FTP

Gridbus Services for eScience applications Application Development Environment: XML-based language for composition of task farming (legacy) applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine. Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes that meet user QoS requirements. Hide Low-Level Grid Middleware interfaces Globus (v2, v4), SRB, Alchemi, Unicore, and ssh-based access to local/remote resources managed by XGrid, Condor, SGE.

Figure 3 : Logging into the portal. Click Here for Demo Figure 3 : Logging into the portal. Drug Design Made Easy!

Deadline (D) and Budget (B) Constrained Scheduling Algorithms Execution Time (D) Execution Cost (B) Compute Grid Data Grid Cost Opt Limited by D Minimize Yes Cost-Time Opt Minimize if possible Time Opt Limited by B Conservative-Time Opt Limited by B, jobs have guaranteed minimum budget

GridSim: A Toolkit for Modeling and Simulation of Various Types of Grids Application, User, Grid Scenario’s Input and Results Application Configuration Resource Configuration Visual Modeler Grid Scenario . . . Output Grid Resource Brokers or Schedulers’s Simulation GridSim Toolkit Application Modeling Resource Entities Information Services Resource Allocation Auction Framework Statistics Add your own policy for resource allocation Resource Modeling and Simulation (with Time and Space shared schedulers) Single CPU SMPs Clusters Load Pattern Network Reservation Basic Discrete Event Simulation Infrastructure SimJava Distributed SimJava Virtual Machine (Java, cJVM, RMI) PCs Workstations SMPs Clusters Distributed Resources

Selected Funded Projects: 2006-2009 Australian Research Council: Discovery Project: QoS-based Scheduling of e-Research Application Workflows on Global Grids QoS-based Workflow Scheduling SLA-based Resource Allocation Dept. of Science, Training and Education International Science Linkage The Utility Grid Project: Autonomic and Utility-oriented Global Grids for Powering Emerging E-Research Applications Links Gridbus Efforts to Many International Efforts

…… Grid Workflow Application Modeling & Definition Tools Grid Workflow Specification and Verification Grid Workflow Management System Resource Info Service Application Catalogue Build Time Run Time Workflow Design & Monitoring Workflow Execution Control & Monitoring Interaction with Grid resources Interaction with VO Info services QoS-based Workflow Scheduling Fault Management Data Management Data Catalogue Virtual Organization R2 Rn …… SLA-based Resource Allocation System (Plug-in for Existing Local Resource Managers) Negotiation Services Execution Monitor E-Researchers/Users feedback Core Grid Services Synchrotron Data source Global Grid

SLA-based Allocation of Cluster Resources Computation Node 1 Job Manager Computation Node c : Economy-based Admission Control Economy-based Resource Allocation Computation Nodes User u Manager Node Node Status Monitor Job Control User Population User 1 job 1 2 3 4 5 Utility-driven Job Scheduler execution results feedback job submission Pricing Resource Manager

InterGrid: Internetworking of Islands of Grids 5th International Conference on Grid and Cooperative Computing Changsha, China, October 2006 InterGrid: Internetworking of Islands of Grids Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.gridbus.org/gridsim

The Outcomes of Grid Computing thus Far: Islands of Grids

InterGrid: An architecture for Internetworking of “islands” of Grids

The InterGrid: How to get there?

New Research Challenges Decentralised “service” discovery from gateways-based to “leaf” nodes Protocols and Business models for “Peering” between Grids Application models that support growing and shrinking of application Resource management and scheduling SLA based resource allocation.

Summary and Conclusion Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications. Delivering ICT services as computing utilities. Australia is actively developing Grid technologies and applications Recently Australian Govt. is pushing Grids heavily under e-Research banner. Melbourne is leading a number of Grid projects nationally. SOA + Market-based Grid Management = Utility Grids Grids allow users to dynamically lease Grid services at runtime based on their quality, cost, availability, and users QoS requirements. Grids offer enormous opportunities for realizing e-Science and e-Business at global level.

Thanks for your attention! We Welcome Cooperation in Research and Development! http:/www.gridbus.org