1. Update Propagation with Variable Connectivity Prasun Dewan Department of Computer Science University of North Carolina dewan@unc.edu

2. 2 Evolution of wireless networks n Early days: disconnected computing (Coda’91) u Laptops plugged in at home or office u No wireless network n Later: weakly connected computing (Coda ‘95) u Assume a wireless network available, but u Performance may be poor u Cost may be high u Energy consumption may be high u Intermittent disconnectivity causes involuntary breaks

3. 3 Issues raised by Coda’91 n Considers strong connection and disconnection u weak connection? hoarding, emulation, or something else?

4. 4 Design Principles for Weak Connectivity n Don’t make strongly-connected clients suffer u E.g. wait for weakly connected client to give token n Weak-connection should be better than disconnection n Do communication tasks in background if possible u user patience threshold n In weakly connected mode, communicate u the more important info u what is important? n When in doubt ask, user n Should be able to work without user advice

5. 5 Disadvantages of Disconnection n Updates not visible to others n Updates at risk due to theft, loss, damage n Conflicts more likely n Cache misses may impede progress n Exhaustion of cache space

6. 6 State Transition Diagram HoardingMergingEmulation Provide access to cached, possibly stale data Track changes to cached data Detect conflicts Resolve conflicts Disconnection Physical Reconnection Logical Reconnection Serve cache miss Hoard walk Replace stale data Write through Write disconnected Weak connection Serve cache miss Mark stale data Background adapted hoard walk Track changes & trickle merging Strong Connection, User command Weak connection

7. 7 Immediate vs. Delayed Service of Cache Miss On cache miss if fetch time < user-patience-threshold get remote object (in foreground) else add to items fetched during next hoard walk (in background) (2 + e 0.01p )

8. 8 Replacing vs. Marking Stale Data n Replacing stale data would make strongly connected clients wait for weakly connected ones. n Stale data replaced by invoking client callbacks n Stale data marked by breaking a client callback u Requires sending of message to client u Only on first remote update to cached object u Client probably sends message to break callback so that server does not have know who is weakly connected u Could give feedback to user, disallow changes or viewing of stale data, but not mentioned in paper

9. 9 Volume Callbacks n Volume callbacks another efficiency mechanism u Broken when any object in a volume changes u Also make merging more efficient F If entire volume is unchanged then individual objects not examined u Can still invoke (any) file callbacks if volume callback broken n Adaptation to fine-grained (Sync-like) merging u Hierarchical callbacks

10. 10 Hoard Walk n Done every u h time units (10 minutes) u or user request n Not the same as connected hoard walk n Stale small files automatically included n User has option to marks large files to be fetched

11. 11 Trickle Merging n Do not propagate local changes immediately u reduces # of messages sent u reduces size of data sent F buffering changes allows log compression F important because whole file sent F what if small (Sync-like) changes sent?

12. 12 Trickle Merging vs. Write-back Caching n Do not propagate local changes immediately u reduce network traffic n Weak consistency u Conflicts n Changes buffered in persistent storage n Buffering time large u Many Minutes, days n Do not propagate local changes u reduce latency n Strong consistency n No conflicts n Changes buffered in volatile memory n Buffering time small n Seconds, few minutes

13. 13 Trickle Merging Implementation write f1mkdir d1 TMt1 < ATM mkdir d2mkdir d3 t1 > A t1 < A

14. 14 Trickle Merging Implementation write f1mkdir d1 TM mkdir d2mkdir d3 Do high priority task (e.g. servicing cache miss)

15. 15 Trickle Merging Implementation mkdir d1 TM mkdir d2mkdir d3rm d1 Committed items not candidate for compression

16. 16 Trickle Merging Implementation mkdir d3rm d1

17. 17 Trickle Merging Implementation n Consider those log changes created > A time units (600 s) ago u Allows change to age u 50% compression effectiveness on all traces n Break these into chunks (C time units) u Amount of data transmitted in time C based on available bandwidth u Urgent messages sent between chunk transmission F Cache miss services n Trickle merging not done atomically u New log changes F not combined with changes chosen at merge start time F not sent for merging n Not specified when initiated u Periodic, during inactivity, when earliest item ages

18. 18 Merging n Times chosen to get desired level of compression n Desired level of consistency?

19. 19 Trading Consistency for Performance Consistency (Divergence) Performance

20. 20 Two methods to control divergence n Rover: Mix pessimistic and optimistic u Worry about W-W conflicts but not R-W conflicts F Some objects may be cached in read-only mode. F Verify object not changed before allowing change Pull model: client can check Push model: server can inform (Coda) u Worry about R-W conflicts. F Expire object after timeout. n Lotus Notes, Suite, Coda:Keep optimistic model u Reduce transaction length F based on real-time F push model F see other ways in TACT

21. 21 TACT: Combining and extending aspects of the two approaches n Optimistic and pessimistic combined u Can choose which conflicts are delayed n Optimistic transaction length bounded u Real-time used as in Rover and Coda u Logical properties of operations also used F Three tunable parameters per replicating host F Can simulate pessimistic, optimistic and points in between. F Control staleness of data read and tentativeness of local writes u Model called continuous consistency

22. 22 Time-based Staleness Staleness of data controlled based on real-time u Pull or push? F Push requires a host’s synchronization times to be known by all remote hosts F How to guarantee staleness level given message delay u TACT local host pulls data from remote host F after it has not heard form the latter t seconds F guaranteed to not see data more stale than t seconds blocks during pull, so message delays do not count performance probably worse u Called staleness error F “Physical Staleness”

23. 23 Content-based Staleness Staleness of data controlled based on how much remote values diverge from local value. u Measuring value divergence? F Could be number of remote writes, but some writes more major than others –allocate five seats vs. 1 seat some writes cancel others –allocate vs. deallocate seat F Sum of (+/-) weights of remote writes alloc n seats given weight n decalloc n seats given weight -n u Local host pull vs. remote hosts push F push because remote hosts knows what changes made

24. 24 Content-based Staleness (contd) u For each host i, one value, e i, defined for all other hosts F conceptually want to think of divergence of value v I at host i from final value with all distributed changes taken into account u Value divided evenly among all hosts F each remote host checks abs(divergence) i < e i /n -1 F if not sends its local changes to host i F thus each remote host processes independently F decentralized process (local decision on each write) F change of parameter requires broadcast and reallocation u e i is called numeric error since it is a number F in case of numeric objects, represents error of local version F but also applicable to non-numeric results bulletin board F represents a numeric value for staleness depending on content of object F logical staleness

25. 25 Tentativeness of Local Writes n Local writes tentative until ordered wrt to remote writes n Parameter determines how many local writes buffered until resolution process occurs u a subset of remote peers may be sent the writes F primary server F quorum n Called order error u assumes merge process simply orders the writes F other approaches possible taking avg, min discarding order not important for commuting operations u In case list insertions, causes insertions to be out of order F bulletin board example

26. 26 TACT Example Host A X = 3 Y = 6 X += 1 Y += 2 X += 2 Y += 4 Host B X = 1 Y = 3 X += 1 Y += 3 Staleness = 3 Tentativeness = 3 Staleness = 8 Tentativeness = 1

27. 27 Simulating Optimistic (Weak Consistency) n No special process used to reduce conflicts u Content-based staleness = infinity u Time-based staleness = infinity u Write tentativeness = infinity u A replica uses regular mechanisms to merge F export in Rover F anti-entropy in Bayou F strong connection in Coda’91

28. 28 Simulating Pessimistic n Each operation checked for global conflict u Content-based staleness = 0 or u Time-based staleness = 0 or u Tentativeness = 0 u of all replicas F A replica syncs with all other replicas on each operation

29. 29 Staleness vs. Tentativeness n Not orthogonal u Reducing staleness reduces tentativeness F implies more frequent synchronization n Not equivalent u Pull vs push u Reducing tentativeness may not reduce staleness F all hosts may not be contacted n Tentativeness = 0 vs Staleness = 0 u In both cases all local allowed changes will be committed in the correct order u But some remote changes may be reordered u Chances of reordering smaller if staleness = 0 u Serialiazability vs Linearizability (Strong consistency)

30. 30 Possible Extensions n Use type-based serializability u for numeric values such as int, real F automatically determine weights u table F automatically determine put(k, e) cancelled by remove(e) put(k1, e1) commutes with put (k2, e2) and remove(k2) –do not have to worry about merging them or order errors among them u sequence F automatically determine insert (i, e) cancelled by del(e) u simulate Coda and Rover weak merging

31. 31 Simulating Trickle Merging n Merge with some period, p u Content-based staleness = infinity u Time-based staleness = p u Write tentativeness = infinity u Not the same, because pull in TACT instead of push in CODA’95