Distributed Systems Introduction

2 CS 403 Distributed Systems D.S. Theory Peer to peer systems Cloud Computing Sensor Networks

The Hype!  Forrester in 2010 – Cloud computing will go from $40.7 billion in 2010 to $241 billion in  Gartner in Cloud computing revenue will soar faster than expected and will exceed $150 billion by It will represent 19% of IT spending by  IDC in 2009: “Spending on IT cloud services will triple in the next 5 years, reaching $42 billion.”  Companies and even Federal/state governments using cloud computing now: fbo.gov

Many Cloud Providers AWS: Amazon Web Services – EC2: Elastic Compute Cloud – S3: Simple Storage Service – EBS: Elastic Block Storage Microsoft Azure Google Cloud Google Compute Engine Rightscale, Salesforce, EMC, Gigaspaces, 10gen, Datastax, Oracle, VMWare, Yahoo, Cloudera And many many more!

Two Categories of Clouds  Can be either a (i) public cloud, or (ii) private cloud  Private clouds are accessible only to company employees  Public clouds provide service to any paying customer: Amazon S3 (Simple Storage Service): store arbitrary datasets, pay per GB-month stored Amazon EC2 (Elastic Compute Cloud): upload and run arbitrary OS images, pay per CPU hour used Google AppEngine/Compute Engine: develop applications within their appengine framework, upload data that will be imported into their format, and run

Reference Texts  Andrew S. Tanenbaum and Marten van Steen: “Distributed Systems: Principles and Paradigms”  Prentice-Hall  George Coulouris, Jean Dollimore and Tim Kindberg: “Distributed Systems: Concepts and Design”  Addison-Wesley

7 Organization of the course 1Characterization of DS 2System Models 3Networking and Internetworking 4Interprocess Communication Foundations 10Time and Global States 11Coordination and Agreement Distributed algorithms 6Operating System Support 8Distributed File Systems 15Distributed Multimedia Systems 16Distributed Shared Memory 18Mach Case Study System infrastructure Middleware 5Distributed Objects and Remote Invocation 7Security 9Name Services 17CORBA Case Study 12Transactions and Concurrency Control 13Distributed Transactions 14Replication Shared data *

8 Can you name some examples of Operating Systems?

9 … Linux WinXP Vista Unix FreeBSD Mac OSX 2K Aegis Scout Hydra Mach SPIN OS/2 Express Flux Hope Spring AntaresOS EOS LOS SQOS LittleOS TINOS PalmOS WinCE TinyOS …

10 What is an Operating System?  User interface to hardware (device driver)  Provides abstractions (processes, file system)  Resource manager (scheduler)  Means of communication (networking)  …

11 Can you name some examples of Distributed Systems?

12 Distributed Systems Examples  Client-server (e.g., NFS)  The Internet  The Web  A sensor network  DNS  BitTorrent (peer to peer overlays)  Datacenters

13 What is a Distributed System?

Definition of a Distributed System  A distributed system is a collection of independent computers that appears to its users as a single system.

A definition (Coulouris, et al)  System of networked computers that  communicate and coordinate their actions only by passing messages  concurrent execution of programs  components fail independently of one another

A definition (Lamport)  “You know you have a distributed system when the crash of a computer you’ve never heard of stops you from getting any work done.” inter-dependencies shared state

Definition of a Distributed System A distributed system organized as middleware. Note that the middleware layer extends over multiple machines. 1.1

Distributed System

Distributed Systems19 Next Generation Information Infrastructure Visualization Battle Planning Visualization Collaborative Multimedia (Telemedicine) Collaborative Task Clients Server farms Battle Planning Electronic Commerce Distance Learning Requirements - Availability, Reliability, Quality-of-Service, Cost-effectiveness, Security DeviceNets & SensorNets Wide Area Network (Internet)

Distributed Systems20 Mobile & ubiquitous distributed systems

Distributed System Example – A Datacenter

Servers Front Back In Some highly secure (e.g., financial info)

Power Off-site On-site WUE = Annual Water Usage / IT Equipment Energy (L/kWh) – low is good PUE = Total facility Power / IT Equipment Power – low is good (e.g., Google~1.11)

Cooling Air sucked in from top (also, Bugzappers)Water purified Water sprayed into air 15 motors per server bank

25 List of Topics we’ve Covered  Distributed system Basics  Lamport timestamps  File system  Name space  Failure detectors  Replication  Clouds and their predecessors (e.g., Grids and timesharing industry)  Sensor networks

Operating System types  Centralized Systems Process management  Network System Share resources Remote access Telnet / FTP No direct control from machine to another  Distributed system Global view of files system Global name Global time ….

Example of Distributed Systems  Network of workstations in university It has a single file system with all files accessible from all machines  Workflow information system (automatic order) Orders are placed by means of laptop that are connected to the system through telephone network  World Wide Web To publish a document by give it unique URL

Distributed Application Examples  Automated banking systems  Tracking roaming cellular phones  Air-traffic control

Example: Embedded Systems  Automotive control systems Mercedes S class cars these days are equipped with 50+ autonomous embedded processors Connected through proprietary bus-like LANs  Consumer electronics Audio HiFi equipment

Sensor Networks Organizing a sensor network database, while storing and processing data (a) only at the operator’s site or …

Sensor Networks Organizing a sensor network database, while storing and processing data … or (b) only at the sensors.

(A) Plants and Animals interacting in the Food Chain Which is a Distributed System – (A) or (B)? (A)

(B) The Internet (Internet Mapping Project, color coded by ISPs) (B)

34 What is a Cloud?  It’s a cluster! It’s a supercomputer! It’s a datastore!  It’s superman!  None of the above  All of the above  Cloud = Lots of storage + compute cycles nearby

35 A Sample Cloud Topology Top of the Rack Switch Core Switch Servers Rack If higher bandwidth link, then a “fat tree” topology

36 Timesharing Industry (1975): Market Share: Honeywell 34%, IBM 15%, Xerox 10%, CDC 10%, DEC 10%, UNIVAC 10% Honeywell 6000 & 635, IBM 370/168, Xerox 940 & Sigma 9, DEC PDP-10, UNIVAC 1108 Grids (1980s-2000s): GriPhyN (1970s-80s) Open Science Grid and Lambda Rail (2000s) Globus & other standards (1990s-2000s) First large datacenters: ENIAC, ORDVAC, ILLIAC Many used vacuum tubes and mechanical relays P2P Systems (90s-00s) Many Millions of users Many GB per day Data Processing Industry : $70 M. 1978: $3.15 Billion. Berkeley NOW Project Supercomputers Server Farms (e.g., Oceano) “A Cloudy History of Time” © IG 2010 Clouds

37 Course Projects 1.Multiparty Democracy and distributed systems 2.Smartphones for healthy living 3.Mobile ad-hoc live streaming 4.Social network 5.Datacenter topology 6.New transport protocol 7.Graph analysis 8.Energy-efficient datacenters 9.DNS 10.VM scheduler 11.Instant Messaging 12.New cloud services 13.Pricing for clouds 14.Health information systems