CS542: Topics in Distributed Systems Diganta Goswami

Distributed System A collection of independent computers that appears to its users as a single coherent system. Autonomous computers Many components – connected by a network – sharing resources.

Distributed System A System of networked components that communicate and coordinate their actions only by passing messages concurrent execution of programs no global clock components fail independently of one another

Another definition You know you have a distributed system when the crash of a computer you’ve never heard of stops you from getting any work done. inter-dependencies shared state independent failure of components

A working definition for us A distributed system is a collection of entities, each of which is autonomous, programmable, asynchronous and failure-prone, and which communicate through an unreliable communication medium using message passing. Entity=a process on a device (PC, PDA) Communication Medium=Wired or wireless network Our interest in distributed systems involves design and implementation, maintenance, algorithmics Designers and progammers interested in : algorithmics, maintenance. (informal definition, for us programmers of distributed systems) Ok: peer to peer systems Contradiction: a computer without ROM or disk drives that needs to boot over the network. Is a collection of these computers a distributed system?

“Important” Distributed Systems Issues No global clock: no single global notion of the correct time (asynchrony) Unpredictable failures of components: lack of response may be due to either failure of a network component, network path being down, or a computer crash (failure-prone, unreliable) Highly variable bandwidth: from 16Kbps (slow modems or Google Balloon) to Gbps (Internet2) to Tbps (in between DCs of same big company) Possibly large and variable latency: few ms to several seconds Large numbers of hosts: 2 to several million

There are a range of interesting problems for Distributed System designers Real distributed systems Cloud Computing, Peer to peer systems, Hadoop, distributed file systems, sensor networks, graph processing, … Classical Problems Failure detection, Asynchrony, Snapshots, Multicast, Consensus, Mutual Exclusion, Election, … Concurrency RPCs, Concurrency Control, Replication Control, … Security Byzantine Faults, … Others…

Typical Distributed Systems Design Goals Common Goals: Heterogeneity – can the system handle a large variety of types of PCs and devices? Robustness – is the system resilient to host crashes and failures, and to the network dropping messages? Availability – are data+services always there for clients? Transparency – can the system hide its internal workings from the users? Concurrency – can the server handle multiple clients simultaneously? Efficiency – is the service fast enough? Does it utilize 100% of all resources? Scalability – can it handle 100 million nodes without degrading service? (nodes=clients and/or servers) Security – can the system withstand hacker attacks? Openness – is the system extensible?

Challenges and Goals of Distributed Systems Heterogeneity Openness Security Scalability Failure handling Concurrency Transparency

Challenges Heterogeneity (variety and difference) of underlying network infrastructure, Internet consists of many different sorts of network – their differences are masked by the fact that all of the computers attached to them use the Internet Protocols for communication. e.g. a computer attached to an Ethernet has an implementation of the Internet Protocols over the Ethernet, whereas a computer on a different sort of network will need an implementation of the Internet Protocols for that network.

Heterogeneity Computer hardware and software e.g., operating systems, compare UNIX socket and Winsock calls Programming languages : in particular, data representations

Some approaches: Middleware A S/W layer that provides a programming abstraction as well as masking the heterogeneity of the underlying networks, H/W, O/S and programming languages. Middleware (e.g., CORBA): transparency of network, hard- and software and programming language heterogeneity. JAVA RMI In addition to solving the problems of heterogeneity, middleware provides a uniform computational model for use by the programmers of servers and distributed applications.

Positioning Middleware General structure of a distributed system as middleware. 1-22

Openness Characteristic that determine whether the system can be extended and re-implemented in various ways. Determined primarily by the degree to which new resource sharing services can be added and be made available for use by a variety of client programs. Cannot be achieved unless the specification and documentation of the key s/w interfaces are made available to s/w developers (i.e. key interfaces are published)

Openness Designers of the Internet protocols introduced a series of documents called RFCs Specifications of the Internet communication protocols Specifications for applications run over them e.g., email, telnet, file transfer, etc. (by the mid 80’s) RFCs are not the only means --- e.g. CORBA is published through a series of documents, including a complete specification of the interfaces of its services (www.omg.org)

Openness Offering services according to standard rules that describe the syntax and semantics of those services e.g., Network protocol rules (RFCs) Services specified through interfaces Interface definition languages (IDLs) specifies names and available functions as well as parameters, return values, exceptions etc.

Security Distributed systems must protect the shared information and resources The openness of DS makes them vulnerable to security threats Providing security is a significant challenge for DS

Security. Integrity: protection against alteration or corruption Privacy / Confidentiality: protection against disclosure to unauthorized individuals Integrity: protection against alteration or corruption Availability: protection against interference with the means to access the resources

Scalability Scalable system—system that can handle additional number of users/resources without suffering noticeable loss of performance Three metrics of a scalable system No of user/resources Distance between the farthest nodes in the system (network radius) Number of organizations exerting control over the pieces of the system

Challenges in designing scalable DS Controlling the cost of physical resources: As the demand for a resource grows, it should be possible to extend the system, at reasonable cost, to meet it. e.g. it must be possible to add server computers to avoid the performance bottleneck that would arise if a single file server had to handle all file access request when the freq. of file access request grows in an intranet with the increase in users and computers. www.amazon.com is more than one computer

Challenges in designing scalable DS Controlling the performance loss: Management of a set of data whose size is proportional to the number of users or resources in the system e.g. the Domain Name System holds the table with the correspondence between domain names of computers and their Internet address Hierarchic structures scale better than linar structures.

An example of dividing the DNS name space into zones. Scaling Techniques 1.5 An example of dividing the DNS name space into zones.

Challenges in designing scalable DS Preventing s/w resources running out: Numbers used as Internet address --- 32 bits was used in the late 70’s but may run out soon. Change from 32 bits to 128 bits? Difficult to predict the demand. Over-compensating for future growth may be worse than adapting to a change when we are forced to - large Internet address occupy extra space in messages and in computer storage.

Failure Handling Failure in a DS is partial Some components fail while others continue to function This makes handling of failures difficult.

Techniques for dealing with failures Detecting failures may be impossible – remote site crash or delay in message transmission? Some can be. Ex. - Checksums can be used to detect corrupted data

Techniques for dealing with failures Masking failure Some can be hidden or made less severe Retransmission – when messages fail to arrive

Techniques for dealing with failures Tolerating failures Would not be practical to detect and hide all of the failures. Can be designed to tolerate some of those e.g. timeouts when waiting for a web resource – clients give up after a predetermined number of attempts and take other actions & inform the user.

Failure Handling Recovery from failures Redundancy Rollback Undo/Redo in transactions Redundancy Makes the system more available through replication of resources/data Redundant routes in the network Replication of name tables in multiple domain name servers

Concurrency In a distributed system it is possible that multiple machines/processes/users may try to access shared data/resource concurrently Can potentially lead to incorrect results and/or Deadlocks The operations must be synchronized/serialized so that the end result is correct

Transparency Concealing the heterogeneous and distributed nature of the system so that it appears to the user like one system Making the user believe that there is only a single, undivided system i.e., to hide the notion of distribution completely What are the challenges of transparency?

Transparency Categories Access transparency - access local and remote resources using identical operations e.g., users of UNIX NFS can use the same commands and parameters for file system operations regardless of whether the accessed files are on a local or remote disk.

Transparency categories Location Transparency: Access without knowledge of location of a resource e.g., URLs, email addresses (hostname, IP addresses, etc. not required --- the part of the URL that identifies a web server domain name refers to a computer name in a domain, rather than to an Internet address)

Transparency Categories Concurrency transparency: Allow several processes to operate concurrently using shared resources in a consistent fashion w/o interference between them. That is, users and programmers are unaware that components request services concurrently. Replication transparency Use replicated resource as if there was just one instance. Increase reliability and performance w/o knowledge of the replicas by users or application programmers.

Failure transparency Enables the concealment of faults, allowing users and application programs to complete their task despite failures of h/w or s/w components. Retransmit of email messages – eventually delivered even when servers or communication links fail – it may even take several days.

Failure transparency Failure transparency depends on concurrency and replication transparency. Replication can be employed to achieve failure transparency Message transmission governed by TCP is a mechanism for providing failure transparency

Mobility Transparency Mobility transparency: allow resources to move around w/o affecting the operation of users or programs e.g., 700 phone number – but URLs are not, because someone’s personal web page cannot move to their new place of work in a different domain – all of the links in other pages will still point to the original page!

Transparency Categories Performance transparency: adaptation of the system to varying load situations without the user noticing it. Scaling transparency: allow system and applications to expand without need to change structure or application algorithms

Degree of transparency There are systems in which attempting to blindly hide all distribution aspects from users is not always a good idea Requesting your electronic newspaper in your mailbox before 7 am local time – while you are at the other end of the world living in a different time zone (Your morning paper will not be the morning paper you are used to)

Degree of transparency There is trade-off between a high degree of transparency and the performance of a system Masking transient server failure by retransmitting the request may slow down the system If it is necessary to guarantee that several replicas need to be consistent all the time, a single update may take a long time – something that cannot be hidden from the user.