1. Grid Computing Research @ Melbourne: Gridbus Perspective
Dr. Rajkumar Buyya
Grid Computing and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne, Australia
Gridbus Sponsors

2. Grids @ Various Schools in Melbourne: Technology Creation -> Applications
Faculty of Engineering – CS/EE/Geomatics
GRIDS Lab and the Gridbus Project – technology focus
SensorWeb (NICTA)
Natural Language Processing
Virtual organisation for tsunami research
Faculty of Science - Physics
Belle and LHC Grid
Australian Virtual Observatory
Faculty of Medicine – HFI, Anatomy
MRI Grid
Distributed Kidney Modelling
Social Sciences/Humanities
The development of a collaborative distributed environment for humanities research projects whose primary data takes the form of ethnographic audiovisual media.
A Grid-enabled technology for the sharing of graphics, audio, video and text between social science research groups working internationally.
Information Tech. Services Division
MARC (Melbourne Advanced Research Computing)/HPC

3. Classes of Grid Services
Computational Services – CPU cycles
Pooling computing power: TeraGrid, AusGrid, ChinaGrid, IndiaGrid, UK Grid,…
Data Services
Collaborative data sharing generated by instruments, sensors, persons: LHC Grid, Napster
Application Services
Access to remote software/libraries and license management—NetSolve
Interaction Services
eLearning, Virtual Tables, Group Communication (Access Grid), Gaming
Knowledge Services
The way knowledge is acquired, processed and managed—data mining.
Utility Computing Services
Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.
Utility Grid
Users
Knowledge Grid
Interaction Grid
ASP Grid
Data Grid
infrastructure
Computational Grid

4. Grid Challenges Application Construction Security
Uniform Access
Security
System Management
Computational Economy
Resource Discovery
Resource Allocation
& Scheduling
Data locality
Network Management
Application Construction

5. + Community Services: e.g., IEEE TC for Scalable Computing
GRIDS Melbourne
R & D
Education
Youngest and one of the rapidly growing research labs in our School/University:
Founded in 2002
Houses:
Research Fellows/PostDocs
Research Programmers
PhD candidates
Honours/Masters students
Funding
National and International organizations
Australian Research Council & DEST
Many industries (Sun, StorageTek, Microsoft, IBM, Microsoft)
University-wide collaboration:
Faculties of Science, Engineering, and Medicine
Many national and international collaborations.
Academics
Industries
Software:
Widely in academic and industrial users.
Publication:
GRIDS Lab produces over 20% of our Dept’s research output.
+ Community Services: e.g., IEEE TC for Scalable Computing

6. Selected Funded Projects: 2004-2006
Australian Research Council:
Discovery Projects:
"The Development of Data Grids for High Energy Physics"
"Economic Scheduling for Efficient Management of Clusters and their Cooperative Federation"
Application Projects:
"Development of e-Research Tools for an MRI Grid Computing Facility"
"e-Research Grid Environment for Integration of Distributed Kidney Models and Resources”

7. Pushes Grid computing into mainstream computing
The Gridbus Melbourne: Enable Leasing of ICT Services on Demand
WWG
Gridbus
Pushes Grid computing into mainstream computing

9. What does Grid players require?
They need tools and technologies that help them in value expression, value translation, and value enforcement.
Grid Service Consumers (GSCs):
How do I express QoS requirements ?
How do I trade between timeframe & cost ?
How do I map jobs to resources to meet my QoS needs?
How do I manage Grid dynamics and get my work done? …
Grid Service Providers (GSPs)
How do I decide service pricing models ?
How do I specify them ?
How do I translate them into resource allocations ?
How do I enforce them ?
How do I advertise & attract consumers ?
How do I do accounting and handle payments?

10. Solution 1: Service Oriented Architecture (SOA)
A SOA is a contractual architecture for offering and consuming software as services.
There are four entities that make up an SOA
service provider,
service registry, and
service consumer (also known as service requestor).
The functions or tasks that the service provider offers, along with other functional and technical information required for consumption, are defined in
the service definition or contract.
registry
contract
provider
consumer

11. Solution 2: Market-Oriented Grid Computing - (a) Sustained Resourced Sharing and (b) Effective Management of Shared Resources
For resource providers, they provide service with different goals
For consumers, they have different requirments
How to manage grid service supply-and-demand is hard task
So how can consumers find a resource depending on their requirements.
And what type of resource that providers provide and how can they serve consumers within certain QoS.
For example, when many consumers want to access same resource, who should be served first
How to manage supply-demand balance is another challenge.
Grid Economy

12. Service-Oriented Grid Architecture
Data Catalogue
Grid Bank
Grid Market Services
Information Service
Sign-on
Health Monitor
Info ?
Grid Explorer
Grid Node N … …
Programming Environments
Secure
Job Control
Agent
Applications
Schedule Advisor
Grid Node1
QoS
Pricing Algorithms
Trade Server
Trading
Trade Manager
Accounting
Resource
Reservation
Misc. services …
Deployment Agent
JobExec
Resource Allocation
Grid Resource Broker
Storage R1 R2 … Rm
Core Middleware
Services
Grid Service Consumer
Grid Service Providers

13. Gridbus and Complementary Technologies – realizing Utility Grid…
Grid
Applications
Science
Commerce
Engineering
Collaboratories
Portals …
MPI
ExcellGrid
Gridscape
Workflow
X-Parameter Sweep Lang.
User-Level Middleware …
Grid Brokers:
Nimrod-G
Workflow Engine
Gridbus Data Broker
Globus
Unicore
Grid Storage Economy
Grid Exchange & Federation
Grid Market Directory
Core Grid
Middleware …
Grid Bank
Alchemi
NorduGrid
XGrid
Grid Economy
.NET
JVM
Condor
PBS
SGE
Libra
Tomcat
Grid
Fabric
Software
Windows
Solaris
Linux
AIX
IRIX
OSF1
Mac
CDB
Grid
Fabric
Hardware
PDB
Worldwide Grid

14. On Demand Assembly of Services: Putting Them All Together
Visual Application Composer
Application Code
Explore data 1
Results+ Cost Info 10
Grid Resource Broker 2
Bill 12
Data Catalogue 5
ASP Catalogue
Grid Info Service
Grid Market Directory 4 6 3
Job 8
Results 9 7
GSP
(e.g., UofM) PE
(e.g., VPAC)
(e.g., IBM)
CPU
or PE
Grid Service (GS)
(Globus)
Alchemi GS
GTS
Cluster Scheduler
GSP
(Accounting Service)
Gridbus GridBank 11

16. Alchemi: .NET-based Enterprise Grid Platform & Web Services
Alchemi Manager
Web Services
Internet
Alchemi Users
Internet
like Model
General Purpose
Dedicated/Non-dedicate workers
Role-based Security
.NET and Web Services
C# Implementation
GridThread and Job Model Programming
Easy to setup and use
Widely in use!
Alchemi Worker Agents

17. Many users in Universities: See next for an example.
Some Users of Alchemi
Tier Technologies, USA
Large scale document processing using Alchemi framework
Satyam Computers Applied Research Laboratory, India
Micro-array data processing using Alchemi framework
CSIRO, Australia
Natural Resource Modeling
The University of Sao Paulo, Brazil
The Alchemi Executor as a Windows Service
stochastix GmbH, Germany
Serving clients in International Banking/Finance sector
The Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland
Patterns of transcription factors in mammalian genes
Many users in Universities: See next for an example.

18. Agenda Introduction Global Grids and Challenges
Utility Networks and Grid Computing
Application Drivers and Various Types of Grid Services
Global Grids and Challenges
Security, resource management, pricing models, …
Service-Oriented Grid Architecture and Gridbus Solutions
Market-based Management, GMD, Grid Bank, Alchemi
Grid Service Broker
Architecture, Design and Implementation
Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids
A Case Study in High Energy Physics
Summary and Conclusion

19. Grid Service Broker (GSB)
A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.
It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation)
Key Features
A single window to manage & control experiment
Programmable Task Farming Engine
Resource Discovery and Resource Trading
Optimal Data Source Discovery
Scheduling & Predications
Generic Dispatcher & Grid Agents
Transportation of data & sharing of results
Accounting

20. Gridbus Broker Architecture
Gridbus Client
Gribus Client
Gridbus Client
(Bag of Tasks Applications)
App, T, $, Opt
(Data Grid Scheduler)
Gridbus Farming Engine
Schedule Advisor
Record Keeper
Trading Manager
Grid Dispatcher
Grid Explorer
Grid Middleware
TM TS $
GE GIS, NWS
Grid Info Server
RM & TS G $
Data
Catalog
Data
Node C $ U G
Unicore enabled node.
Globus enabled node. L A
RM: Local Resource Manager, TS: Trade Server
Alchemi enabled node.

21. Gridbus Broker: Separating “applications” from “different” remote service access enablers and schedulers
Application Development Interface
Home Node/Portal
Gridbus
Broker
fork()
batch()
-PBS
-Condor
-SGE
-Alchemi
-XGrid
Single-sign on security
Alogorithm1
Scheduling Interfaces
AlogorithmN
Data Catalog
Plugin Actuators
Alchemi
-PBS
-Condor
-SGE
Globus
Job manager
fork()
batch()
Gridbus
agent
Data Store
Gateway
Unicore
-PBS
-Condor
-SGE
-XGrid
SSH
fork()
batch()
Gridbus
agent
Access Technology
SRB
Grid FTP

22. Gridbus Services for eScience applications
Application Development Environment:
XML-based language for composition of task farming (legacy) applications as parameter sweep applications.
Task Farming APIs for new applications.
Web APIs (e.g., Portlets) for Grid portal development.
Threads-based Programming Interface
Workflow interface and Gridbus-enabled workflow engine.
Resource Allocation and Scheduling
Dynamic discovery of optional computational and data nodes that meet user QoS requirements.
Hide Low-Level Grid Middleware interfaces
Globus (v2, v4), SRB, Alchemi, Unicore, and ssh-based access to local/remote resources managed by XGrid, Condor, SGE.

23. Figure 3 : Logging into the portal.
Click Here for Demo
Figure 3 : Logging into the portal.
Drug Design
Made Easy!

24. Deadline (D) and Budget (B) Constrained Scheduling Algorithms
Execution Time (D)
Execution Cost (B)
Compute Grid
Data Grid
Cost Opt
Limited by D
Minimize
Yes
Cost-Time Opt
Minimize if possible
Time Opt
Limited by B
Conservative-Time Opt
Limited by B, jobs have guaranteed minimum budget

26. GridSim: A Toolkit for Modeling and Simulation of Various Types of Grids
Application, User, Grid Scenario’s Input and Results
Application Configuration
Resource Configuration
Visual Modeler
Grid
Scenario
. . .
Output
Grid Resource Brokers or Schedulers’s Simulation
GridSim Toolkit
Application Modeling
Resource Entities
Information
Services
Resource Allocation
Auction Framework
Statistics
Add your own policy for resource allocation
Resource Modeling and Simulation (with Time and Space shared schedulers)
Single CPU
SMPs
Clusters
Load Pattern
Network
Reservation
Basic Discrete Event Simulation Infrastructure
SimJava
Distributed SimJava
Virtual Machine (Java, cJVM, RMI)
PCs
Workstations
SMPs
Clusters
Distributed Resources

27. Selected Funded Projects: 2006-2009
Australian Research Council:
Discovery Project:
QoS-based Scheduling of e-Research Application Workflows on Global Grids
QoS-based Workflow Scheduling
SLA-based Resource Allocation
Dept. of Science, Training and Education
International Science Linkage
The Utility Grid Project: Autonomic and Utility-oriented Global Grids for Powering Emerging E-Research Applications
Links Gridbus Efforts to Many International Efforts

28. …… Grid Workflow Application Modeling & Definition Tools Grid Workflow
Specification and Verification
Grid Workflow Management System
Resource Info Service
Application Catalogue
Build Time
Run Time
Workflow Design & Monitoring
Workflow Execution Control & Monitoring
Interaction with Grid resources
Interaction with VO Info services
QoS-based Workflow Scheduling
Fault Management
Data Management
Data Catalogue
Virtual Organization R2 Rn ……
SLA-based Resource Allocation System
(Plug-in for Existing Local Resource Managers)
Negotiation Services
Execution
Monitor
E-Researchers/Users
feedback
Core Grid Services
Synchrotron
Data source
Global Grid

29. SLA-based Allocation of Cluster Resources
Computation Node 1
Job Manager
Computation Node c :
Economy-based Admission Control
Economy-based Resource Allocation
Computation Nodes
User u
Manager Node
Node Status Monitor
Job Control
User Population
User 1
job 1 2 3 4 5
Utility-driven Job Scheduler
execution results
feedback
job submission
Pricing
Resource Manager

30. InterGrid: Internetworking of Islands of Grids
5th International Conference on Grid and Cooperative Computing
Changsha, China, October 2006
InterGrid: Internetworking of Islands of Grids
Rajkumar Buyya
Grid Computing and Distributed Systems
(GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne, Australia

31. The Outcomes of Grid Computing thus Far: Islands of Grids

32. InterGrid: An architecture for Internetworking of “islands” of Grids

33. The InterGrid: How to get there?

34. New Research Challenges
Decentralised “service” discovery from gateways-based to “leaf” nodes
Protocols and Business models for “Peering” between Grids
Application models that support growing and shrinking of application
Resource management and scheduling
SLA based resource allocation.

35. Summary and Conclusion
Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications.
Delivering ICT services as computing utilities.
Australia is actively developing Grid technologies and applications
Recently Australian Govt. is pushing Grids heavily under e-Research banner.
Melbourne is leading a number of Grid projects nationally.
SOA + Market-based Grid Management = Utility Grids
Grids allow users to dynamically lease Grid services at runtime based on their quality, cost, availability, and users QoS requirements.
Grids offer enormous opportunities for realizing e-Science and e-Business at global level.

36. Thanks for your attention!
We Welcome Cooperation in Research and Development!