26-Oct-15COMP28112 Lecture 171 COMP28112 – Lecture 17 Grid and Cloud Computing Where are we? We covered all key material. After the break, we’ll talk about.

Slides:



Advertisements
Similar presentations
1 From Grids to Service-Oriented Knowledge Utilities research challenges Thierry Priol.
Advertisements

A Dynamic World, what can Grids do for Multi-Core computing? Daniel Goodman, Anne Trefethen and Douglas Creager
What is Grid Computing? Cevat Şener Dept. of Computer Engineering, METU.
High Performance Computing Course Notes Grid Computing.
Amazon. Cloud computing also known as on-demand computing or utility computing. Similar to other utility providers like electric, water, and natural gas,
By Adam Balla & Wachiu Siu
Clouds from FutureGrid’s Perspective April Geoffrey Fox Director, Digital Science Center, Pervasive.
What is Cloud Computing? o Cloud computing:- is a style of computing in which dynamically scalable and often virtualized resources are provided as a service.
Milos Kobliha Alejandro Cimadevilla Luis de Alba Parallel Computing Seminar GROUP 12.
Undergraduate Poster Presentation Match 31, 2015 Department of CSE, BUET, Dhaka, Bangladesh Wireless Sensor Network Integretion With Cloud Computing H.M.A.
Presented by Sujit Tilak. Evolution of Client/Server Architecture Clients & Server on different computer systems Local Area Network for Server and Client.
An Introduction to Cloud Computing. The challenge Add new services for your users quickly and cost effectively.
Plan Introduction What is Cloud Computing?
CLOUD COMPUTING. A general term for anything that involves delivering hosted services over the Internet. And Cloud is referred to the hardware and software.
© 2008 by M. Stümpert, A. Garcia; made available under the EPL v1.0 | Access the power of Grids with Eclipse Mathias Stümpert (Karlsruhe Institute.
Cloud Computing الحوسبة السحابية. subject History of Cloud Before the cloud Cloud Conditions Definition of Cloud Computing Cloud Anatomy Type of Cloud.
A Brief Overview by Aditya Dutt March 18 th ’ Aditya Inc.
Distributed Systems Early Examples. Projects NOW – a Network Of Workstations University of California, Berkely Terminated about 1997 after demonstrating.
Cloud Computing Saneel Bidaye uni-slb2181. What is Cloud Computing? Cloud Computing refers to both the applications delivered as services over the Internet.
PhD course - Milan, March /09/ Some additional words about cloud computing Lionel Brunie National Institute of Applied Science (INSA) LIRIS.
TRANSFORMING IT.  The cloud is a group of servers.  A user interacts with the cloud without worrying about how it is implemented. Cloud computing describes.
Cloud Computing 1. Outline  Introduction  Evolution  Cloud architecture  Map reduce operation  Platform 2.
Software Architecture
DISTRIBUTED COMPUTING
CS525: Special Topics in DBs Large-Scale Data Management Hadoop/MapReduce Computing Paradigm Spring 2013 WPI, Mohamed Eltabakh 1.
Grid Computing - AAU 14/ Grid Computing Josva Kleist Danish Center for Grid Computing
Hadoop/MapReduce Computing Paradigm 1 Shirish Agale.
Cloud Computing in NASA Missions Dan Whorton CTO, Stinger Ghaffarian Technologies June 25, 2010 All material in RED will be updated.
Cloud Computing A set of Internet-based application.
Presented by: Mostafa Magdi. Contents Introduction. Cloud Computing Definition. Cloud Computing Characteristics. Cloud Computing Key features. Cost Virtualization.
1 © 2009 Cisco Systems, Inc. All rights reserved.Cisco Confidential Cloud Computing – The Value Proposition Wayne Clark Architect, Intelligent Network.
| nectar.org.au NECTAR TRAINING Module 3 Common use cases.
What is the cloud ? IT as a service Cloud allows access to services without user technical knowledge or control of supporting infrastructure Best described.
Grid Middleware Tutorial / Grid Technologies IntroSlide 1 /14 Grid Technologies Intro Ivan Degtyarenko ivan.degtyarenko dog csc dot fi CSC – The Finnish.
Ames Research CenterDivision 1 Information Power Grid (IPG) Overview Anthony Lisotta Computer Sciences Corporation NASA Ames May 2,
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
Authors: Ronnie Julio Cole David
GRID Overview Internet2 Member Meeting Spring 2003 Sandra Redman Information Technology and Systems Center and Information Technology Research Center National.
1 Cloud Computing Advisor : Cho-Chin Lin Student : Chien-Chen Lai.
1 Melanie Alexander. Agenda Define Big Data Trends Business Value Challenges What to consider Supplier Negotiation Contract Negotiation Summary 2.
CS525: Big Data Analytics MapReduce Computing Paradigm & Apache Hadoop Open Source Fall 2013 Elke A. Rundensteiner 1.
7. Grid Computing Systems and Resource Management
| nectar.org.au NECTAR TRAINING Module 4 From PC To Cloud or HPC.
Securing the Grid & other Middleware Challenges Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer.
3/12/2013Computer Engg, IIT(BHU)1 CLOUD COMPUTING-1.
Web Technologies Lecture 13 Introduction to cloud computing.
Hadoop/MapReduce Computing Paradigm 1 CS525: Special Topics in DBs Large-Scale Data Management Presented By Kelly Technologies
HPC in the Cloud – Clearing the Mist or Lost in the Fog Panel at SC11 Seattle November Geoffrey Fox
Toward a common data and command representation for quantum chemistry Malcolm Atkinson Director 5 th April 2004.
3/12/2013Computer Engg, IIT(BHU)1 CLOUD COMPUTING-2.
1 TCS Confidential. 2 Objective : In this session we will be able to learn:  What is Cloud Computing?  Characteristics  Cloud Flavors  Cloud Deployment.
Directions in eScience Interoperability and Science Clouds June Interoperability in Action – Standards Implementation.
PARALLEL AND DISTRIBUTED PROGRAMMING MODELS U. Jhashuva 1 Asst. Prof Dept. of CSE om.
Distributed Geospatial Information Processing (DGIP) Prof. Wenwen Li School of Geographical Sciences and Urban Planning 5644 Coor Hall
1 Views of Cloud Computing Prof. Ravi Sandhu Executive Director and Endowed Chair March 25, © Ravi Sandhu.
Grid and Cloud Computing
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING CLOUD COMPUTING
Dr.S.Sridhar, Director, RVCT, RVCE, Bangalore
Clouds , Grids and Clusters
An Introduction to Cloud Computing
Dr.S.Sridhar, Director, RVCT, RVCE, Bangalore
Recap: introduction to e-science
Week 01 Comp 7780 – Class Overview.
Grid and Cloud Computing
COMP28112 – Lecture 19 The Grid (and Cloud Computing)
COMP28112 – Lecture 17 The Grid (and Cloud Computing) Where are we
Some problems and more…
COMP28112 – Lecture 19 The Grid (and Cloud Computing) Where are we
Grid and Cloud Computing
Presentation transcript:

26-Oct-15COMP28112 Lecture 171 COMP28112 – Lecture 17 Grid and Cloud Computing Where are we? We covered all key material. After the break, we’ll talk about some open-ended/advanced topics/problems and the exam!

26-Oct-15COMP28112 Lecture 172 An Early Experiment The advent of the Internet made scientists think about the possibility of exploiting interconnected machines for time- consuming applications. E.g.: Long Integer Factorisation (find the prime factors of an integer – remember: public-key cryptography relies upon the difficulty of finding the prime factors of long integers): –Algorithms for integer factorisation might be time consuming: for integers whose factors are two primes of about the same size, no polynomial time algorithm (to find the factors) is known. –

26-Oct-15COMP28112 Lecture 173 An Early Experiment (cont.) Around 1990, the Internet mostly consists of Unix machines. A C program (implementing an integer factorisation algorithm) was developed that would run on a machine when it was idle; it would use to communicate with a server, to results, to request data: –At a time when factoring 100-digit-long integers would take one month using expensive machines, it was realized that with a good implementation such integers could be factored within a few days and for free! –Read “factoring by electronic mail”, EUROCRYPT1989

26-Oct-15COMP28112 Lecture 174 HOME (download and analyse radio telescope data)

26-Oct-15COMP28112 Lecture 175 Is all this (number crunching) useful? Science (and human progress) rely on curiosity… Frank Nelson Cole found in 1903 the prime factors of 2 67 –1: –2 67 –1 = × –How long did it take him? “Three years on Sundays”. How long would it take today? (read: “In Praise of Science: Curiosity, Understanding, and Progress”)

26-Oct-15COMP28112 Lecture 176 The origins of the Grid The term ‘Grid’ was coined around –It was used to describe a hardware and software infrastructure that was needed by the rapidly growing and highly advanced community of High Performance Computing (HPC): HPC refers to the use of (parallel) supercomputers and computer clusters linked together in a way that provides high-end capabilities; the latter are needed for a number of scientific applications (‘big science’). WARNING: This was a vision!

26-Oct-15COMP28112 Lecture 177 What is the vision of the Grid? “A computational Grid is a hardware and software infrastructure that provides dependable, consistent, pervasive and inexpensive access to high-end computational capabilities” (“The Grid: Blueprint for a New Computing Infrastructure”,1998) Why ‘Grid’? –Electricity Grid: “A network of high-voltage transmission lines and connections that supply electricity from a number of generating stations to various distribution centres, so that no consumer is dependent on a single station”. –As early as 1969, it was suggested that: “We will probably see the spread of computer utilities, which, like present electric and telephone utilities, will service individual homes and offices across the country”.

26-Oct-15COMP28112 Lecture 178 So, what is the Grid? The original term was ‘catchy’ –Soon, researchers started talking about: Data Grids Knowledge Grids Access Grids Science Grids Bio Grids Sensor Grids Campus Grids Tera Grids Commodity Grids, and so on… The sceptic would wonder if there was more to the Grid than a ‘funding concept’.

26-Oct-15COMP28112 Lecture 179 A Grid checklist The Grid coordinates resources that are not subject to centralized control … (i.e., resources of different companies, or different administrative domains) … using standard, open, general-purpose protocols and interfaces… … to deliver non-trivial qualities of service (for example, related to response time, throughput, availability, security, …)

26-Oct-15COMP28112 Lecture 1710 Defining the Grid… There are competing definitions, defining it as: –An implementation of distributed computing –A common set of interfaces, tools, APIs, … –The ability to coordinate resources across different administrative domains (creating Virtual Organisations) –A means to provide an abstraction (virtualisation) of resources, services, … –Resource sharing and coordinated problem solving in dynamic virtual organisations

26-Oct-15COMP28112 Lecture 1711 Grid computing must provide… Resource discovery and information collection and publishing Data Management on and between resources Process Management on and between resources Common Security Mechanisms Process and Session Recording/Accounting

26-Oct-15COMP28112 Lecture 1712 Some middleware available… Globus ( –GridFTP (extensions to FTP protocol to cope with HPC and Grid Security) –OGSA-DAI ( standard approach for data access on the Grid –WS-Resource Framework: allows the use of established Web Services standards (slow, complex, …). Condor, Condor-G, HTCondor ( –Workload Management for compute-intensive jobs

26-Oct-15COMP28112 Lecture 1713 Some Grid Infrastructures (and beyond the Grid…) Teragrid, now Extreme Science and Engineering Digital Environment ( – Datagrid ( – UK National Grid Service ( Many more national initiatives… Learn about the grid: Beyond the Grid: –GLORIAD: –PlanetLab:

26-Oct-15COMP28112 Lecture 1714 Grid Computing Summary There is a lot of hype around grid computing… …but there is real-world value in e-science, e- business… …through virtualisation of the underlying distributed resources! The Large Hadron Collider Grid has been set up to support the Large Hadron Collider experiment: – There has been related research in the School

26-Oct-15COMP28112 Lecture 1715 Cloud Computing Largely evolved from the Grid. The idea: –On-demand resource provisioning. Users of the Cloud are consumers who don’t have to own the resources they use; these resources are provided by others (providers) as a service! Associated concepts: –Software as a service (SaaS) –Platform as a service (PaaS) –Infrastructure as a service (IaaS)

26-Oct-15COMP28112 Lecture 1716 Virtualisation: the key concept In the same way that resources of a single computer are virtualised by traditional OSs, a cloud computing layer can virtualise multiple computers. Hardware Operating System App Traditional View Hardware OS App Virtualisation OS Virtualized View Hardware

26-Oct-15COMP28112 Lecture 1717 Grid & Cloud Similarities: Scalability, Divide and Conquer, Lots of Data, SLAs, … Differences: The Grid has a tighter relation between users and administrators; the execution model on the Grid is more restrictive. A subject for debate: – – The Cloud has raised privacy concerns. For an example of a Cloud Computing service: Amazon’s EC2: Grids and (even more) Clouds are here to stay!

26-Oct-15COMP28112 Lecture 1718 The Applications… Often, these are scientific workflows consisting of sequences of tasks. Tasks may be partitioned into parallel tasks each of which operates on different data (a technique known as divide-and-conquer) Tools are available to enable the automatic composition of the workflows and their mapping onto resources – Applications in astronomy, earth sciences, bioinformatics, …

26-Oct-15COMP28112 Lecture 1719 Remember in Lecture 2?

Dealing with large data sets Speed-up processing by parallelizing the operations! Map-Reduce Framework –Apache Hadoop: open source implementation The ‘Big Data’ fashion/obsession/buzzword… –Data + Framework + Algorithms = Knowledge? – 30TB of data every night 1 petabyte of data per second 26-Oct-15COMP28112 Lecture 1720

A challenge for the future… The huge amounts of data we can collect and process will offer us great opportunities to improve quality of life and shape progress in a way that benefits everybody but to achieve this we need a better understanding of the challenges in handling and analyzing large-scale data including new technologies… 26-Oct-15COMP28112 Lecture 1721

Finally… Motivating some advanced problems (but first some statistics about lab 2)

(89150) Messages to the server based on hour of the day (0-1, 1-2, …, 23-24)

(120853) Messages to the server based on hour of the day (0-1, 1-2, …, 23-24)

What day of the week was busiest? (1: Mon, 7: Sun)

What day of the week was busiest? (1: Mon, 7: Sun) 26-Oct-15COMP28112 Lecture 1726

Messages since 9/2/2014 (=day 1) 4 th March 23 rd April

Messages since 9/2/2015 (=day 1) 26 Feb

The problems 1.When is it best to execute two tasks on one machine as opposed to one task on one machine + communication (data transfer) and computation on another? 2.How to design random values, which are not uniform? E.g., two- digit integer where the probability is: 0-9 (90%) and (10%) 3.How to compute the average of streamed data? (related to lab exercise 3, where you have to compute the average over large values) 4.Split the following loop, so that each of 4 threads takes the same amount of computation. Rewrite, in terms of t, where t is between 0 and 3. for(i=0;i<1000;i++) { some_computation_on_i; } Marking sessions for the labs: May 5 th & 8 th All your lab work should be marked by May 8!