A CSEL presentation based on I. Foster, Z. Yong, I. Raicu, and S. Lu, "Cloud Computing and Grid Computing 360-Degree Compared," in Grid Computing Environments Workshop, GCE '08, 2008, pp

Overview Grid and Cloud Comparison Business Model Architecture Resource Management Programming Model Application Model Security Model 2

3 To meet computational needs Single supercomputer is too expensive A distributed system Coordination of existing resources (e.g. computer clusters) to form virtual supercomputer Mostly used in universities and government laboratories 1 hard-working person VS many average person Parallelizing jobs

Do you know what exactly is Cloud computing? Born as a relative to Grid computing and Utility computing Cluster computing Distributed systems Moving computation on PC to centrally managed resources By oneself - private Cloud By third-party – public Cloud (e.g. Amazon) Hybrid 4

“In the mid 1990s, the term Grid was coined to describe technologies that would allow consumers to obtain computing power on demand. Ian Foster and others posited that by standardizing the protocols used to request computing power, we could spur the creation of a Computing Grid, analogous in form and utility to the electric power grid.” I. Foster, Z. Yong, I. Raicu, and S. Lu, "Cloud Computing and Grid Computing 360-Degree Compared," in Grid Computing Environments Workshop, GCE '08, 2008, pp

Resources: compute resource, file (data) exchange, software and others. Virtual Organization (VO): organisations usually with common interest contribute to (and thus form) the Grid 6 “…coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations.” I. Foster, "The Anatomy of the Grid: Enabling Scalable Virtual Organizations," in Proceedings of the 7th International Euro-Par Conference Manchester on Parallel Processing, 2001, pp. 1-4.

7 The Grid provides high computation capacity and data storage through the coordination of shared distributed resources. Economies of scale Driven by the need for higher computation capacity and storage while reducing costs, organizations share their resources - thus forming the Grid environment - in exchange for access to the resources owned by other organizations inside the Grid S. Smanchat, "Scheduling Parameter Sweep Workflow in the Grid," PhD, Caulfield School of Information Technology, Monash University, Australia, 2012.

8 Enabling ways to solve problem that were not possible Enables organisations to perform the tasks that require computation power exceeding their own by delegating tasks to be executed by others. Increasing resource utilization Other parties can make use of any idle resources in the Grid according to resource sharing agreement S. Smanchat, "Scheduling Parameter Sweep Workflow in the Grid," PhD, Caulfield School of Information Technology, Monash University, Australia, 2012.

9 Ground concept remains similar: reducing cost, increasing reliability and flexibility by using federated resources Context changes: Big Data Virtualization Clusters are expensive Virtualization of commodity servers (or even clusters) is a better choices Problems in the resource management also remains similar. Usage Discovery Programming

10 “A large-scale distributed computing paradigm that is driven by economies of scale, in which a pool of abstracted, virtualized, dynamically- scalable, managed computing power, storage, platforms, and services are delivered on demand to external customers over the Internet.” I. Foster, Z. Yong, I. Raicu, and S. Lu, "Cloud Computing and Grid Computing 360-Degree Compared," in Grid Computing Environments Workshop, GCE '08, 2008, pp

11 Massively scalable Abstract entity No complicated setup is required (unlike Grid) Different levels of services (e.g. packages) for end users: Economies of scale Services can be re-configured and delivered on demand.

12 Cloud is a distributed system Cloud evolves from Grid computing And may use Grid computing as supporting infrastructure Utility computing Computing resources are metered (e.g. utility Grid) Service-Oriented Architecture (SOA)

I. Foster, Z. Yong, I. Raicu, and S. Lu, "Cloud Computing and Grid Computing 360-Degree Compared," in Grid Computing Environments Workshop, GCE '08, 2008, pp

14 G. Mateescu, W. Gentzsch and C. J. Ribbens, "Hybrid Computing—Where HPC meets grid and Cloud Computing," Future Generation Computer Systems, vol. 27, pp , 2011.

15 “Grid is a system that 1) Coordinates resources that are not subject to centralized control 2) Using standard, open, general-purpose protocols and interfaces 3) To deliver nontrivial qualities of services” I. Foster, "What is the Grid? - a three point checklist," GRIDtoday, vol. 1, However, central administration control may be present in some Grids Globus Toolkit Version 4 (de facto standard)

16 Cloud Usage-based payment (e.g. EC2 and S3) VS traditional one-time payment for software Access 100,000 cores without full investment Grid Resource sharing (e.g. CPU cycle) within VO Project-oriented Usually used in academic institutions Give and take

17 I. Foster, "The Anatomy of the Grid: Enabling Scalable Virtual Organizations," in Proceedings of the 7th International Euro-Par Conference Manchester on Parallel Processing, 2001, pp. 1-4.

18 Fabric layer – access to resources E.g. Condor, GARA (General architecture for advanced reservation) Connectivity layer – communication and authentication GSI (Grid Security Infrastructure) Resource layer – protocols related to resources GridFTP, GRAM (Grid Resource Access and Management) Collective layer – monitor and discover VO resources E.g. Condor-G, Nimrod-G Application layer – whatever you want E.g. Grid workflow, Grid portal Globus Toolkit covers the first four layers

19 I. Foster, Z. Yong, I. Raicu, and S. Lu, "Cloud Computing and Grid Computing 360-Degree Compared," in Grid Computing Environments Workshop, GCE '08, 2008, pp

20 Fabric layer – hardware resources Unified resource layer – abstract resources through virtualization Platform layer – specialized services providing development platform E.g. web hosting Application layer – applications on Cloud

21 IaaS – Infrastructure as a Service Hardware (virtual) and software for application environment Scalable (and reconfigurable) dynamically E.g. EC2 and S3 PaaS – Platform as a Service Higher level environment for deployment of application E.g. Google’s App Engine SaaS – Software as a Service Specialized software accessed remotely through the Internet e.g. Office Web Apps

22 G. Sakellari and G. Loukas, "A survey of mathematical models, simulation approaches and testbeds used for research in cloud computing," Simulation Modelling Practice and Theory, 2013.

23 AWS Console as of August 2014

24

25 Grid compute model Batch-scheduled model Users submit job > request resource > wait for resource > run Usually long wait queue and data staging Hardware-bound queuing system Cloud compute model Resources are shared by all users at the same time in the Cloud via virtualization No hardware-bound queuing

26 Cloud data model Everything on the Cloud VS Ian Foster’s vision Can we trust Cloud security? Do we want to do things offline? Listening to music on the Cloud? Desktop supercomputer I. Foster, Z. Yong, I. Raicu, and S. Lu, "Cloud Computing and Grid Computing 360-Degree Compared," in Grid Computing Environments Workshop, GCE '08, 2008, pp

27 Grid data model Data Grid E.g. management of data replicas GridFTP - analogous to BitTorrent Virtual data – data abstraction Location transparency Distributed metadata catalog Privacy and access control Materialization transparency (transfer VS recompute)

28 Other concerns Data locality File system and data transfer Merging compute and data management Moving data around for computation Virtualization Monitoring Provenance Log history of execution Implemented in workflow management systems

29 MPI (Message Passing Interface) – most commonly used For multi-core architecture / cluster MPICH-G2 - Grid-enabled MPI with Globus Toolkit integration Grid programming focuses on management of large numbers of datasets and tasks. The motivation for Grid workflow system MapReduce – distributes program to data Integration and interoperability are the challenges in the Cloud

30 Grid HPC / HTC applications MPI Workflow applications Loosely coupled applications Homogeneous / heterogeneous tasks of small / large sizes Cloud Not well-defined yet Transaction-oriented Interactive

31 Grid Hardware are shared by different institutions Likely to be heterogeneous resources Each site has its own security measure Single sign-on GSI (Grid Security Infrastructure) based on PKI Cloud Hardware usually belong to single organization Likely to be homogeneous resources Simpler and less secure (please check for update) Most important for Cloud’s success

32 Data only accessed by the authorized users Security certificate of Cloud provider Data (physical) location and privacy Data segregation – separate data of each user Recovery Support for investigation Data viability, even though the provider is taken over

33 Random interesting scenario