1. Software Upgrades for Distributed Systems
Sameer Ajmani
Google
Shadow methods similar to Haskell monads? In that they relate types.
Barbara Liskov
MIT
Liuba Shrira
Brandeis

2. Internet Services Are long-lived, robust Run on many machines
Must be continuously available
Have persistent state
Face ever-changing requirements
Require software upgrades to
Fix bugs
Improve performance
Add/change/remove features
Long-lived implies robust to failures!

3. Upgrade Requirements Automatic, Controlled Deployment
Ensure continuous availability
Test new software on a few nodes
Upgrade servers before clients
Mixed mode operation
Upgrades may be slow due to transforms v1 v3 v2 v2
(upgrading) v2

4. Outline System & Upgrade Model Specifying Upgrades
Implementation Models

5. System Model A node is an object of class C
Different nodes may run different classes
Nodes communicate via RPCs
Examples of classes include storage servers, routers, clients
Can be extended to more general models (e.g., multiple objects per physical node).

6. Upgrade Model
A class upgrade replaces an old class, Cold, with a new one, Cnew
Implements types Told, Tnew
May be compatible or incompatible
An upgrade is a set of class upgrades
We focus on incompatible upgrades here.

7. Supporting Mixed Mode
Each node handles calls to past and future versions of itself
Adding support for new versions must be fast
Removing support for old versions should be easy
We chose to have receivers handling simulation; callers are oblivious to receiver’s version.
Nodes are responsible for their own behavior.
We considered doing simulation on the clients -- doesn’t really work.
TRANSITION: We need some software structure to support this!

8. Simulation Objects
Each node handles calls to past and future versions of itself C3 F5 F4 P2
DESCRIBE the diagram:
- Every node is running a current version.
- This node is running at version 3, but supports versions 2 through 5. Version 3 is the highly-optimized actual software; versions 4, 5, and 2 are simulated via other objects. Version 1 is no longer supported.
- There needs to be some mechanism for dispatching calls; we’ll discuss this later.
- The arrows are one possible way of connecting nodes.
Future SO Installed when a new version is introduced
Past SO Installed when the node upgrades
Future SOs simulate future behavior
Past SOs simulate past behavior
Adding/removing an SO does not require a restart

9. Simulation Objects
Each node handles calls to past and future versions of itself C3 F5 F4 P2
Internal node upgrades (e.g., performance); not interface change -> no SOs
Compatible upgrades; need future SO, but not past SO
Incompatible upgrades: need both future and past SO -- this is our focus
SOs are only required for certain upgrades

10. Specifying Upgrades
Shadow methods similar to Haskell monads? In that they relate types.

11. Specifying Upgrades Must behave like a single object
Even when upgrades are incompatible
Upgrade specification must define this
Goal: no surprises for clients!
E.g., changing permissions to ACLs
This is the big contribution.
Compatible upgrades are specified implicitly by the subtype relationship.

12. Constraints on Specifications
Type requirement
A call of version V behaves according to the specification of type TV
This requirement is stated differently than in the paper (paper mentions which class implements which type, which should not be specified here).
To a node running v3, the whole world appears to be running v3 -- simplifies writing software!
Calls are stamped with client version. v1 v3 v3 v3 v2 v1 v2

13. Constraints on Specifications
Sequence requirement
Each event must reflect all earlier ones despite:
Client upgrades
Server upgrades
Version introduced
Version retired
Explain version retirement.
File system example: client expects to see files it wrote (or peers wrote) despite these events.
Client upgrade is interesting -- what does the client expect to see on the server after it upgrades?

14. Example ColorSet  FlavorSet
Incompatible upgrade
ColorSet methods: insertColor(x, c), getColor(x), …
E.g., { (1, red), (2, blue), (3, red) }
FlavorSet methods: insertFlavor(x,f), getFlavor(x), …
This is simplistic but analogous to real examples like adding properties to files in a filesystem.

15. Specifications: Invariant
Invariant I relates the object states
I(Oold, Onew)
Oold
Onew I
Choose the weakest invariant possible.
In this case, lots of nondeterminism.
Invariant must be TOTAL over all legal Oold, Onew states.
I: {x|<x,c> in OCS} = {x|<x,f> in OFS}

16. Specifications: Mapping Function
Mapping function MF defines initial state
Onew = MF(Oold) s.t. I(Oold, Onew)
Oold
Onew MF
MF establishes the invariant.
Could have done something more sophisticated here -- we had choices.
Used to create the Future SO when it is installed.
OFS = MF(OCS) = {<x,grape>| <x,c> in OCS}

17. Relating Behavior Only for mutators Onew O’new ? I I Oold O’old m
We thought we could infer the ‘?’ from the invariant and MF and spec of m, but we cannot (as show in following slides).
Only for mutators

18. Relating Behavior Shadow methods relate behavior Told.m  Tnew.$m
Onew
O’new $m I I
Oold
O’old m
Shadow methods relate behavior
Told.m  Tnew.$m
Tnew.p Told.$p

19. Shadow Method Specification
void ColorSet.$insertFlavor(x, f)
Effects: no <x,c> in thispre 
thispost = thispre U {<x,blue>}
Also ColorSet.$delete, FlavorSet.$insertColor, FlavorSet.$delete
Defining shadow methods is usually simple.

20. The Compound Type
I, MF, and the shadow methods define a compound type Told&new
All the methods, with extended specs for mutators
We would like:
Told&new is a subtype of Told, Tnew
When this doesn’t work:
Weaken invariant I
Upgrade scheduling
Disallow methods
FlavoSet&ColorSet is a subtype each of FS and CS
TRANSITION: now how to implement this
Consider introducing compund type before I, MF, shadow in a future talk.

21. Implementation Models

22. Direct Model SO handles calls only to its own version
And delegates down the chain C2 F4 F3 P1
v3 calls
v2 calls
v2 calls
Future: terminology: delegation vs. forwarding
v3 calls

23. Problems with Direct Model
Poor expressive power
E.g., FlavorSet SO doesn’t know about C.insertColor call
Synchronization
From the point of view of F4, calls to other objects break encapsulation. C2 F4 F3 P1
v3 calls
v2 calls
v2 calls
v3 calls

24. Interceptor Model Newest SO gets all calls
And delegates down the chain C1 F4 F3 F2
v3,2,1
calls
v2,1
calls
v1 calls
TRANSITION: what happens when we upgrade to v2, which is incompatible?
all calls

25. Interceptor Model After first (incompatible) upgrade is installed C2
v3,2,1
calls
v2,1
calls
v2 calls
(somewhat different notation from the paper)
Past so is “P2” because it was defined in the upgrade for v2, but it handles v1 calls!
TRANSITION: what happens when we upgrade to v3, which is compatible?
all calls

26. Interceptor Model
After second (possibly compatible) upgrade is installed C3 F4 P3
v3,2,1
calls
v3 calls
This upgrade is compatible, but still need past SO to support version 1!
In this model, there is always at most 1 past SO.
This past SO disappears when version 1 is retired
all calls

27. Interceptor Model Evaluation
Excellent expressive power
Future and past SO must do more
Can reuse code and delegate C3 F4 P3
TRANSITION: we’ve implemented a prototype…
v3,2,1
calls
v3 calls
all calls

28. Prototype Implementation: Upstart
C++ and Sun RPC
Intercepts socket(), read(), write()
Imposes minimal overhead

29. Summary Upstart is the first complete approach
Allows mixed mode operation
The first definition of what must be specified for incompatible upgrades
A powerful and useful implementation model
A prototype implementation

30. Software Upgrades for Distributed Systems
Sameer Ajmani
Google
Barbara Liskov
MIT
Liuba Shrira
Brandeis

31. Disallowing Constraint: never disallow methods of the current object
Future SO may disallow Tnew methods
Past SO may disallow Told methods
In either case, disallow
Mutators whose shadows are problem
Observers that expose problems

32. Some shadows cannot be implemented via delegation
Disallow methods that have unimplementable shadows
Add shadow method to delegate via dynamic updating
Allowed iff T + shadow is a subtype of T
E.g., can’t add delete() to GrowSet
Implement shadow method in interceptor
Impacts transform function
Won’t work for past SOs because of retirement

33. What does Google do? Extensible protocols
Assume defaults for missing fields
Ignore unexpected fields
Round-robin upgrades among replicas
Datacenter-by-datacenter
Extensible protocols are not perfect, e.g., adding do_not_overwrite bit to Store(key, value) method:
Assuming bit is true is correct, but ignoring this bit is dangerous.

34. END OF SLIDES
The remaining slides are leftover from previous talks and may contain stale information

35. Code Execution Call contains version number Called node dispatches F5
Delete this slide (move after last slide).
v5 calls

36. Upstart A system that supports upgrades And a methodology
Joint work with
Barbara Liskov
Liuba Shrira

37. Class Upgrade New and old classes Cnew, Cold Scheduling function SF
Implement Tnew, Told
Scheduling function SF
Transform function TF
Simulation classes Snew, Sold
Might be incompatible
Tnew is not a subtype of Told

38. Class Upgrade Replaces an old class, Cold with a new one, Cnew
Every node running old class will switch to new class eventually
Upgrade is a set of class upgrades

39. Defining an Upgrade Upgrader enters new upgrade at UDB
nodes
. . .
Upgrader enters new upgrade at UDB
Defines a new version

40. Propagating an Upgrade
UDB
nodes
. . .
Nodes query the UDB periodically
Version numbers flow on all messages

41. Executing an Upgrade If upgrade affects the node Runs the SF
And simulates the future
Shuts down, restarts, runs TF
Starts up “normally”
And simulates the past

42. Disallowing Example GrowSet  IntSet For the future SO:
Disallow IntSet.delete
For the past SO:
Disallow GrowSet.isIn
Told&new becomes <Tfuture , Tpast>

43. What Disallowing Provides
Tfuture is a subtype of Told
And it implements Tnew
Tpast is a subtype of Tnew
And it implements Told

44. Transform Functions Implement the identity map
May need to use future SO, create past SO
Must be restartable
Cannot make remote calls

45. Scheduling Functions Can consult the UDB Examples: Rolling upgrade
Big flip
Fast reboot

46. Implementing Upgrades
Need to provide SOs, TF, SF
For the SOs, need an implementation model
See paper for information on TFs and SFs

47. Summary of Specifications
Specification defines the compound type Told&new
I, MF, and the shadows
If the compound type isn’t a subtype, disallow

48. Specifications: Shadow Methods
Shadow methods relate behavior
Told.m  Tnew.$m
Tnew.p Told.$p
Oold
Onew m $m
e.g., FlavorSet.$insertColor

49. Talk Outline Upgrade requirements Upstart overview Specifying upgrades
Implementing upgrades

50. Requirement: Generality
Support for arbitrary changes
Incompatible upgrades
Old features are no longer supported

51. Requirement: Continuous Availability
Service is required 24/7
Even when upgrading
Therefore systems upgrade gradually
Implies mixed-mode operation

52. Requirement: Controlled Deployment
Systems upgrade gradually
But with control
Manual control is impractical
An automatic system
But upgrader needs control

53. Requirement: Persistence
Systems store important state for users
It cannot be lost
But may need to be transformed

54. Requirement: Ease of Use
Avoiding feature creep helps
Upgrader needs to understand only a few recent versions