Distributed Systems Fall 2010 Course introduction

Fall 20105DV0203 Defining distributed systems “A distributed system is one in which components located at networked computers communicate and coordinate their actions by passing messages.” (Coulouris, Dollimore, Kindberg, 2005) ”A distributed system is one in which the failure of computer you didn't even know existed can render your own computer unusable.” (Leslie Lamport, 1987)

Fall 20105DV0204 Outline Staff presentation Course presentation Lessons from last year This year's course Basics and challenges of distributed systems The big assignment

Fall 20105DV0205 Staff presentation Daniel Henriksson Lars Larsson Questions about the assignments? –Send to Questions about lectures? –Send to the appropriate teacher!

Fall 20105DV0206 Course presentation Students should obtain: –Knowledge of theoretical models for distributed systems –Knowledge of problems and solutions in designing and in the implementation of distributed systems

Course presentation The course covers: –Architectural models of distributed systems –Client-Server, peer-to-peer, transactions, transparency, naming, error handling, resource management, and synchronization … and much more! –Computer security in a broad perspective –Distributed programming and middlewares Fall DV020

Fall 20105DV0208 Course presentation Theoretical part (4.5 ECTS) –Theory, methods, algorithms, and principles Practical part (3 ECTS) –Practical obligatory assignments

Fall 20105DV0209 Lessons from last year (positive) Students were very happy with the level of assistance on the assignment Students acquired a lot of new knowledge The average grade given to the course was “very good” Students were very happy with the practical assignment – hard and rewarding

Fall 20105DV02010 Lessons from last year (negative) Common pitfall: students start with the assignment too late Split opinions regarding what to do with extra time during lectures Very limited computer access due to other course Unclear how level 1 was easier than level 2 on the assignment Too large/annoying post-course evaluation!

Fall 20105DV02011 This year's course Added an assignment on Java RMI Note that any programming language may be used for the second, big assignment Let’s vote regarding our policy on what to do about extra time during lectures! Department hopes that computer access will be better this year (but don’t count on it) Clearer info about the big assignment New evaluation form/procedure Keep up good work

5DV02012 Extra time during lectures We want to be able to adopt the schedule to fit your needs Some lectures are somewhat “sparse” If our in-class discussion of topics take longer time, that’s fine (within reason) If not, we can start covering upcoming topics to have more time during denser lectures Consider topics a reading guide, not a guarantee If this poses a problem, please give us feedback, and we will try to remedy the situation 125DV020Fall 2010

5DV02013 What to learn? Book is dense with information –See reading guide on web page – it is actually accurate –Extremely good, but no easy read Start now! You will be busy later... Understand the problems and solutions –Learn the general ideas of algorithms and how/why they work, not every minute step Definitions are very important!

Fall 20105DV02014 Basics of distributed systems Architectural model –Placement of components across the network and their relationships, and the network itself System model –Client/Server HTTP, FTP, … countless others! –Peer-to-Peer (P2P) BitTorrent, Freenet, Direct Connect, … –Combinations: multiple servers, mobile code, mobile agents, thin clients

Fall 20105DV02015 Design of distributed systems Performance issues –Responsiveness –Throughput –Load balancing Quality of Service (QoS) –Reliability, security, performance (time-critical) –Adaptability

Design of distributed systems Caching and replication –Web caching and Content Distribution Networks Dependability –Correctness, security, and fault tolerance Fall DV020

Fall 20105DV02017 Distributed systems challenges Heterogeneous systems –Networks, hardware, OS, languages, protocols Openness Security Scalability –Control physical resource costs and performance loss, resource conservation, bottleneck avoidance

Distributed systems challenges Failure handling –Detection, masking, tolerance, recovery, redundancy Concurrency –No global time, simultaneously running processes Transparency –Access, location, concurrency, replication, failure, mobility, performance, scaling Fall DV020

Fall 20105DV02019 Distributed systems challenges How are parameters passed and how is data converted? –Marshalling, unmarshalling How are distributed resources (functions, methods, objects) published and discovered? How are errors handled? Security! Memory handling – distributed garbage collection?

Fall 20105DV02020 Middleware Distributed systems often utilize middleware to aid development Offers layer of abstraction Extends upon traditional programming models: –Local procedure call → Remote procedure call –OOP → Remote Method Invocation –Event-based programming model

Fall 20105DV02021 Middleware Applications, Services RMI, RPC Request/Reply protocol Marshalling, Unmarshalling UDP, TCP Middleware

Fall 20105DV02022 Operation invocation Data structures must be “flattened” and serialized for transport –External formats, e.g. XML, JSON, Java Object Serialization,... Use interface –Procedures having either input, output, or both –No pointers –Service interface: provided services –Remote interface: operations accessible from other processes –Cross-language/platform interfaces: IDL, WSDL

Fall 20105DV02023 Semantics (Local call = exactly once) Maybe once –Omission failures (lost packets, crashes) At-least-once –Crash failures, arbitrary failures (multiple executions) –Used by Sun RPC At-most-once –Executed exactly once or not at all –Used by Java RMI, Corba

Fall 20105DV02024 Security Distributed system = increased exposure Client- and Server-authentication Client authorization –Is the client allowed to perform X? Proof of execution –Server must be able to prove that something has been executed –Also, non-repudiation: it should not be possible to claim that something did not happen if it did

Fall 20105DV02025 Distributed systems: a mess! Communication performance variations –Latency (delay), bandwidth (throughput), jitter (variation in time) Clocks and timing –Clock drift Interaction models –Asynchronous, synchronous Event ordering –Delays cause replies to arrive to some process before the request

Distributed systems: a mess! Failures –Distributed systems are much more likely to fail unexpectedly –Lost packets, bit errors, local failures, no response, method does not exist, etc … If you can write stable programs in spite of these difficulties, you are a great programmer! Fall DV020

Fall 20105DV02027 The small assignment Previously a group exercise Solved individually You teach yourself Java RMI Deadline next Friday (Sept 10) 13:15 Session for (final) questions that very same day

Fall 20105DV02028 The big assignment Group communication middleware Presentation of working implementation at the end of the course Solved in pairs (2 students per team) Pair up among yourselves before Friday! Deadlines: Design report: September 17 13:15 Whole system: October 26 13:15 Presentations: October 27 – 29

Fall 20105DV02029 The big assignment Apply concepts from theory –Group handling –Message ordering –(Reliable) Multicast of messages –Not security, however

Fall 20105DV02030 Rules and grading Done in pairs 3 levels –Bonus points for the exam (if non- bonus points give you ≥ 30p of 60p total)! Valid for this year's exams only –Level 1: basic system (no bonus) –Level 2: + dynamic groups (3p bonus) –Level 3: + tree-based reliable multicast (6p bonus)

Levels You may change level at any time Level 1 is easiest, but in practice only if you aim for it from the beginning Many problems can be avoided due to greatly lowered fault-tolerance of the system Fall DV020

Fall 20105DV02032 Constraints May use any programming language and any tools you like –...as long as they do not provide a too big advantage (check with us!) –Currently, we can only help with Java RMI –Do not use plain sockets All normal rules apply –Thou shall not cheat, etc.

Fall 20105DV02033 Test and debug application(s) Test application –A user-level application that shows the functionality of the system Debug application –Used to demonstrate the correctness of your implementation These programs can, and likely will, be one and the same! –But make the debug parts non-essential to use the application –Must be GUI applications!

Fall 20105DV02034 Deliverables Deliverable 1 (project plan) –Your interpretation of the assignment –Requirement analysis –Project and time plan –Basic design of the system Yes, really Deliverable 2 (report) –Refers back to Deliverable 1 –Describe your system –...the usual –Make something to be proud of! One of your biggest projects during your time here at CS

Fall 20105DV02035 Live demonstration You will demonstrate your system to us at the end of the course –Written test protocol

Fall 20105DV02036 Good luck! Students have done this before, and succeeded –Certainly not easy –Hard work, big payoff –All students that attempted the entire assignment passed! Hints –Start on time. No really. –Read the whole specification –We know it’s long, but it helps you

Fall 20105DV02037 Next lecture Time, and the lack thereof –No global time, and its implications –Algorithms for (roughly) synchronizing processes