Introduction

Readings r Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edn. 3 m Note: All figures from this book

Who Can Take This Class? r Anyone with some interest in distributed systems and data management r Pre-requisite: m Undergraduate OS (or undergraduate networking course) r This course does have programming and you may have to be willing to learn new technologies

Course Structure r Lectures m Papers will be assigned to you to review m You are expected to participate in class r Three assignments m Programming m Group work r Survey paper and presentation

Course Readings r No official textbook m Lectures are based on (mostly) research papers m UWO has on-line subscriptions to many of the conferences and journals that the papers are found in

A typical portion of the Internet

A Typical Intranet

Portable and Handheld Devices in a Distributed system

Services and the Internet r Increasingly we are seeing more of our applications moving from the PC to the Internet e.g., m – gmail, yahoo m Photo management – Picasso, Kodak, Shutterbug m Word processing – Google apps r Why? m Less work on the user’s behalf m Maybe the potential for less cost for the user

Data Centers r To support this move from the PC to the “Internet” requires a large number of servers, storage, network support etc; m Companies like Amazon, Google, eBay are running data centers with tens of thousands of machines r To make users trust these systems requires that a number of issues be addressed e.g., failure handling

Reasons r Share data and information despite geography m Examples: Internet: Wikipedia, , web sites Intranet: Networked file systems r Availability m Hardware can fail: power outage, disk failures, memory corruption m Software can fail: bugs, mis-configuration, m To achieve high availability requires replications of data/computation with automatic failovers r Aggregate resources of many computers m CPU: Dryad, MapReduce m Bandwidth: Akamai CDN, BitTorrent

Challenges r Consistency m Sharing data consistently among multiple readers/writers r Fault tolerance m Keeping system available despite node or network failures r Security m Authenticating clients or servers m Defending against or audit misbehaving servers

Challenges r System design m Right interface or abstraction? r Implementation m Maximize concurrency m Identify bottlenecks m Reduce load on the bottleneck resource

Case Study: Distributed File System r A distributed file system provides shared access for multiple clients to a shared storage transparently. m NFS is one of the earliest examples r NFS Design m A file server stores all data and handles requests from clients

Case Study r Consistency m When a client edits a shared file when do other clients see the results? r Fault Tolerance m How to keep the system running when the file server is down? Replicate data at multiple servers m How to update replicated data? m How to fail-over among replicas m How to maintain consistency?

Case Study r Security m Adversary can manipulate messages How to authenticate? m Adversary may compromise machines Can the FS remain correct despite a few compromised nodes r Implementation m The file server should serve multiple clients concurrently m Server threads need to modify shared state Avoid race conditions

Other Examples r It’s not just file servers r What about companies like Amazon? Facebook? Google?

Summary r We briefly discussed a high-level view of data distributed r In this course we will discuss various architectures and algorithms and how they are used by major companies like Amazon, Google, etc;