1. Accessing Remote Sites CS 188 Distributed Systems January 13, 2015

2. Deutsch's “Seven Fallacies of Distributed Computing”
1. The network is reliable
2. There is no latency (instant response time)
3. The available bandwidth is infinite
4. The network is secure
5. The topology of the network does not change
6. There is one administrator for the whole network
7. The cost of transporting additional data is zero
Bottom Line: true transparency is not achievable

3. Introduction Distributed systems require nodes to talk to each other
How do they do that, mechanically?
To achieve various effects?
Synchronization of computations
Data passing
File systems

4. Messages The ultimate answer to all of these problems
At the bottom, it’s all done with messages
But messages are very general
So even given you’re using messages, there are many options

5. Key Characteristics of Messages
They are explicitly sent
Someone asks to send them
They have well defined contents
Under control of the sender
They are all-or-nothing
Entire message is delivered or nothing is
Receipt is usually optional
Receiver must ask to get the message
Always unidirectional, usually unicast

6. Synchronizing Computation With Messages
for (i=0; i< 10; i++) {
for (j=0; j< 10; j++) {
/* do stuff */
. . . }
Tell machine B that loop is done
Done!
Machine A
Machine B

7. Conceptually Simple Messages can’t be received before they’re sent
So receiving process can’t start computation till after the sender sends the message
Sender doesn’t send until he’s ready for receiver to go ahead

8. Tricky Issues Delivery isn’t reliable
What if the message doesn’t get there?
Delivery speed is not predictable
So what? But . . .
Assumes receiver follows the rules
Doesn’t start till he gets the message
Does receive the message some reasonable time after its delivery
Security issues

9. What If the Sender Needs Feedback?
Once B starts, do next task
for (i=0; i< 10; i++) {
for (j=0; j< 10; j++) {
/* do stuff */
. . . }
Tell machine B that loop is done
Done!
I started
This will require a second message
Machine A
Machine B
From B to A, this time

10. So What? There is now a round trip delay A to B Then B to A
Two opportunities for messages to get lost
Two chances for member processes to screw up

11. So Why Wait? Machine A Machine B Done! Once B starts, do next task
for (i=0; i< 10; i++) {
for (j=0; j< 10; j++) {
/* do stuff */
. . . }
Tell machine B that loop is done
Done!
Machine A
Machine B

12. Results of Not Waiting Less delay No more two round trip delays
No certainty about event ordering
Did A start its new code before or after B got the message?
What if messages are lost, B fails, etc.?

13. Lessons From Operating System Synchronization
Ensuring proper ordering of events is critical to predictable behavior
Thinking about proper synchronization is hard
Getting proper synchronization usually requires blocking
With bad performance implications
And possible deadlocks

14. Distributed System Implications
Total correctness will be hard and expensive to achieve
Getting acceptable results without total correctness is desirable
For complex scenarios, likely to be many possible outcomes
Some likely to be unpredictable

15. Data Passing With Messages
Many distributed computations require local processes to share data
Often with remote processes
Requires moving data to the other machine
Again, all you’ve got is messages

16. Moving Data Machine A Machine B Seems simple enough But . . . Alpha
Beta
Gamma .
Omega
Alpha
Beta
Gamma .
Omega
Seems simple enough
Machine A
But . . .
Machine B

17. Some Complications What if the data won’t fit in one message?
We must use multiple messages
Leading to issues of ordering, losses, knowing when you’re done, etc.
What if the receiver doesn’t have room to store all of it?
Are we keeping a copy at the sender, as well?
If so, are both copies writeable?
If so, what happens if someone writes?

18. File Systems With Messages
Like data passing, in most ways
But the inherent persistence of the data raises issues
Particularly for data that can be written
Either on the original site or the accessing site

19. Accessing Files With Messages
Someone has file X open
Open file x for read
File X
Open file X
Now what?
Machine A
Machine B

20. Either machine can fail at any time
Reading
Note a few things
Someone has file X open
Read file x
File X
Read file X
Either machine can fail at any time
The data is still on machine A
But there’s a copy on machine B
Maybe more than one
Machine A
Machine B

21. What about machine A’s original copy?
Writing
Someone has file X open
Read file x
Write file x
File X
Write file X
What about machine A’s original copy?
Machine A
Machine B

22. Some Obvious Complexities
What if the write message is lost?
What if A reads the file before the write arrives?
But after B does the write?
Is that good, bad, or ambiguous?
What if A writes the file after B reads it, but before the write message arrives?
There are many other complexities

23. Some Security Issues
A message arrives purporting to come from machine X
Perhaps you would do the requested action for machine X, but . . .
Is it really machine X asking?
If machine X is asking for something, is it really what the message says?
If you send a response, how can you be sure only machine X gets it?

24. Basic Solution Authenticate the message
Obtain evidence that the message came from its purported sender
And that it wasn’t altered in transit
Don’t take actions without suitably strong authentication

25. Message Authentication
In rare cases, the network is closed
E.g., it’s a direct wire to machine X
Nobody else can inject the message, so it must come from X
More commonly, use cryptographic methods
Which we won’t cover in detail here
But typically require secure key distribution

26. One Unhandled Issue
“If you send a response, how can you be sure only machine X gets it?”
Authentication doesn’t help here
If you’re sending out secret information, it isn’t enough that X asked
Only X should see the answer
Usually handled by encrypting the message
With same key issues as above

27. One Other Security Issue
Can an attacker prevent a given message from being received?
A problem of denial of service
Often achieved by causing congestion
Cryptography doesn’t help here

28. Basic Data Transport Machine A asks machine B for some data
In general, more than can be held in one message
How do we handle that?

29. In distributed systems, it rarely does
A Simple Approach
Data Item X
X (Part 1)
Data Item X
X (Part 2)
X (Part 3)
X (Part 4)
Send me X
Assuming everything goes well . . .
Machine A
Machine B
In distributed systems, it rarely does

30. One Possible Problem Machine A Machine B Data Item X
X (Part 1)
Data Item X
X (Part 2)
X (Part 3)
X (Part 4)
Send me X
How do we know something bad happened?
Machine A
Machine B
What remedial action can we take?

31. Options For Lost Messages
Detect loss and cancel operation
Detect loss and request retransmission
Go ahead without the lost data
Ensure redundancy in data sent and regenerate lost data on that basis
Proper choice depends on system specifics

32. Another Possible Problem
X (Part 1)
Data Item X
X (Part 2)
X (Part 3)
X (Part 4)
Send me X
How to handle out-of-order delivery?
Machine A
Machine B

33. Options for Misordered Delivery
Wait
And, possibly, wait, and wait, and wait
How long do you wait?
Assume misordered message was dropped
And take an action appropriate for drops
If it comes in eventually, ignore it
On detection, ask the sender to retransmit

34. Another Potential Problem
What if X is big?
Really big
Perhaps machine A doesn’t know how big
What’s really going on at machine A?

35. Handling the Incoming Data
Data must be stored somewhere
In RAM, at first
Perhaps on disk/flash later
As each message holding part of X arrives, its content must be stored
Machine A must allocate buffer space for that data
How much, for how long?

36. Illustrating the Problem
Send me X
X, part 7
X, part 5
X, part 8
X, part 9
X, part 6
X, part 3
X, part 4
X, part 1
X, part 2
Now what?
Buffer for X
Machine A

37. Another Wrinkle The problem is actually worse at the system level
Every incoming message must be put in a buffer
In the OS or device driver or elsewhere
Stays in the buffer until the application process “receives” it
What if the messages arrive too fast and fill all buffers?

38. What Are Your Options?
When you have used up either the application or system buffers
The same as in the network
Store until you run out of storage
Send it back to where it came from
Drop something
You can make things better by asking for help, though

39. Asking For Help IP doesn’t ask for help
It must deal with each packet on its own
At a higher level, though, help is possible
Flow control
Ask the sender to slow down

40. Where Do We Do Flow Control?
Almost always end-to-end
But “end” has a flexible definition
Could be “end machine”
TCP flow control
Could be “end application”
Application level flow control

41. A Brief Diversion Why not flow control in IP?
Based, in part, on the end-to-end argument
An argument in network and system design
Put application functionality into the application end points, not the network
E.g., if you need flow control for your application, put it in the application
Important caveat – IF the endpoints can actually do it

42. Returning to the Core Problem
We are receiving data
We need to store (at least temporarily) the data we receive
How do we make sure we can do so?
Ideally without wasting resources

43. Possible Answers
Figure out how much data will be sent before you set aside space
So you definitely have enough space
“Use” some of the data as you go along
Allowing you to reuse buffer space
Allocate more space as you need it
Tell sender to stop when you can’t hold more data

44. Making Life Even Worse
The problems we’ve seen occur with only two participating nodes
One is a wonderful magic number in distributed systems
Everything easier when there’s only one of whatever we’re considering
Two is a fairly wonderful magic number
It’s either here or there, which simplifies life
Three and above aren’t wonderful or magic
They make life hard

45. One Example Machine A Machine B Machine C File X Read file X

46. What about Machine C’s copy?
So Far, So Good, But . . .
File X
Write file X
Machine A
Machine B
What about Machine C’s copy?
Machine C

47. Some Choices Send a message to C invalidating its copy
Either from A or B
Send a message to C updating its copy
Don’t allow the situation at all
E.g., don’t allow C to read X
Don’t worry about it
It’s C’s problem, if C cares

48. Implications of This Choice
Making sure everyone knows what’s happening requires more messages
Which could be delayed, lost, etc.
Generally more processing and delay
Not worrying about the state of other nodes requires fewer messages
But those nodes may act on old data
Could get weird behaviors
Preventing problem from occurring reduces system’s utility and power
But at low cost and with fewer surprises

49. Which Choice Is Best? It all depends
What is your model of system behavior?
How important is consistency and what are the effects of inconsistency?
Remember to consider failure models
What if some message is lost/delayed?
Different distributed systems deal with the issue in different ways

50. Another Complexity
“When sorrows come, they come not single spies, but in battalions”
William Shakespeare, Hamlet, Act 4, Scene 5
Distributed systems suffer that problem, too
You can’t be sure there’s only one fault
Multiple independent or related faults are possible
And will happen, sooner or later
Will your solution to one be sabotaged by another?

51. Tying It Back To Distributed Systems
Even the relatively simple issue of moving data around proves complex
Highly desirable to understand how your system behaves
Transparent solutions are nice
But often too expensive
An important design choice