Joint Advanced Students School 2005 Replication and synchronization techniques in the context of mobile applications By Alexander Stage Software Engineering betrieblicher Informationssysteme (sebis) Ernst Denert-Stiftungslehrstuhl Lehrstuhl für Informatik 19 Institut für Informatik TU München wwwmatthes.in.tum.de JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 1

Agenda Motivation Mobile Computing Synchronization Techniques Replication Techniques in Databases Oracle Lite 10g – A Mobile Database Solution Peer to Peer Mobile Computing Future Applications and Developments JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 2

Motivation – A Scenario The disconnected sales-person problem: a typical day Mobile Device Local DB synchronize Mobile Device Local DB replicate Company Central DB Mobile Device Local DB update no communication link travelling path Customer A Customer B Customer C JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 3

Agenda Motivation Mobile Computing Synchronization Techniques Replication Techniques in Databases Oracle Lite 10g – A Mobile Database Solution Peer to Peer Mobile Computing Future Applications and Developments JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 4

Mobile Computing - Characteristics Mobile devices: Laptops Palmtops Smart cell phones Requirements: Data access: Anywhere Anytime Nomadic users Constraints: Limited ressources Variable connectivty: Performance Reliability JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 5

Mobile Computing – Communication Models Client Server Model Peer to Peer Model Client Client Client/ Server Client/ Server Server Client/ Server Client/ Server Client Client JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 6

Two Types of Synchronization Process synchronization Data synchronization System A System B Process A Process B modification Process Process Time Data item 1 modification modification Synchronization Data Item 1 Data Item 1 Process A correspondance modification Data item 1 Time 1: 1 Synchronization 1: 1 Synchronization Main System Process B modification Data Item 1 Data item 1 JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 7

Aspects of Synchronization Amount of participating system: 1:1 1:n Amount of exchanged data: Incremental Usage of timestamps, attached to data items Comparison of sub data structures and exchange Full Conflict resolution mechanisms Questions: who wins ? Merging System time synchronization in distributed systems JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 8

Merging Data During Synchronization: The Product - Sales Example System A System B System A System B Order 50 units of product A Order 25 units of product A Order Product A: 50 Order Product A: 25 Stock Product A: 100 Stock Product A: 100 Stock Product A: 50 Stock Product A: 75 No communiation Apply updates Main System Main System Stock Product A: 25 Stock Product A: 100 Reduce stock, Insert order Reduce stock, Insert order No conflict, minimum stock is preserved JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 9

Synchronization solutions File system oriented: Unix tool rsync: Incremental exchange of files Coda file system Version control systems: CVS WEBDav protocol: HTTP based file transfer protocol Locking and Versioning included Mobile Application oriented: Microsoft Active Sync: Synchronization between Desktop and Windows CE mobile devices JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 10

Standardization Efforts in Data Synchronization: SyncML SyncML defines client server based protocol for synchronization SyncML is XML based, main elements: SyncMessage Sync Commands: Modification commands: add, copy, delete, put and replace Query commands: read and search Response commands: status and results Execution specific commands: sequence and atomic Synchronization models: Two way sync Slow sync One way sync Refresh sync Server-alerted sync JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 11

SyncML Architecture Source: SyncML Representation Protocol, version 1.0.1, http://www.syncml.org/docs/syncml_represent_v101_20010615.pdf JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 12

Agenda Motivation Mobile Computing Synchronization Techniques Replication Techniques in Databases Oracle Lite 10g – A Mobile Database Solution Peer to Peer Mobile Computing Future Applications and Developments JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 13

What is Replication ? Usage of multiple copies of server system (replicas) Goals: Higher availability Better performance Areas of usage: Distributed Systems and Databases Replicated Server Only Ressource Server 1 Server n ........ Replicated Ressource and Server Ressource n Ressource 1 ........ Server n Server 1 JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 14

Replication in Databases Replication of the resource replicating the database Elimination of single point of failure Single server behaviour Correctness criterion: one-copy serializability Synchronization in databases: replication Constant trade-off between consistency and efficiency Update anywhere, anytime, any way replication scheme Simple case: Only a few replica nodes Low transaction rates Small amount of deadlocks and reconciliation System scale up: Problem complexity grows drastically at cubic rates Inconsistent replicas and system delusion (no way to repair the inconsistencies) JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 15

Replication in Databases – A Classification Scheme Classification dimensions: Where Dedicated replica: Single Master (primary master): primary and secondaries Every replica: Multi Master (update everywhere) When Synchronous (eager replication) Asynchronous (lazy replication) Other (not considered here) Interaction activities (network traffic) Constant or linear growth Transaction termination: Voting or Non-Voting techniques JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 16

Abstract Functional Replication Model Distinction between different replication solutions: Phase ordering Phase skipping Request Phase: Client sends a request to one or all replicas Server Coordination Phase: Replicas coordinate their work with each other (ordering of Execution steps) Execution Phase: Replicas execute operation Agreement Coordination Phase: Replicas ensure that they all have done the same thing (2pc) Client Response Phase: Client receives a response JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 17

Synchronous Single Master Replication I Characteristics Keeps all replicas synchronous in consistent state No update anomalies Reduces performance Increase response times Possible solution Trigger on selected tables Updates remotely all replicas Fails if one replica fails Usage of 2pc Implication: No control over when replication should take place Only databases with 2pc support JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 18

Synchronous Single Master Replication II (primary) (secondary) (secondary) Source: Wiesmann, Pedone, Schiper, Kemme, Alonso: Understanding Replication in Databases and Distributed Systems, Proceedings of 20th International Conference on Distributed Computing Systems (ICDCS'2000) JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 19

Asynchronous Single Master Replication I Characteristics Response after primary processed request: Primary takes care about transaction ordering Drawback: stale data items versions Performance is main goal Possible solutions Database system log based propagation: Committed updates only Restriction: Wait for current transactions Further improvement: only send data fields Log table: Trigger records update and insert actions Periodically recreate log table Send log table to replicas (preserving oder) JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 20

Asynchronous Single Master Replication II (primary) (secondary) (secondary) Source: Wiesmann, Pedone, Schiper, Kemme, Alonso: Understanding Replication in Databases and Distributed Systems, Proceedings of 20th International Conference on Distributed Computing Systems (ICDCS'2000) JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 21

Handling Failures Two types of failure: Replica fails: Secondary System not affected Secondary catch up Primary Re-election of primary Most up-to-date replica: common system state Possible data loss Network partition Problem: Which partition is allowed to process further requests ? Majority consensus algorithms Quorum consensus algorithms Danger: Dramatic system performance loss JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 22

Synchronous Multi Master Replication I Similar to single master synchronous replication Updates allowed at every replica JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 23

Synchronous Multi Master Replication II Source: Wiesmann, Pedone, Schiper, Kemme, Alonso: Understanding Replication in Databases and Distributed Systems, Proceedings of 20th International Conference on Distributed Computing Systems (ICDCS'2000) JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 24

Asynchronous Multi Master Replication I Characteristics Multi version concurrency control mechanisms Conflicts Reconciliation Automatic Manually Possible solution Unique timestamps Concatenation of local replica system clock to replica identifier Thomas Write Rule: Update applied if timestamp numerical greater than local timestamp Also known as optimistic replication JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 25

Asynchronous Multi Master Replication II (primary) (primary) (primary) Source: Wiesmann, Pedone, Schiper, Kemme, Alonso: Understanding Replication in Databases and Distributed Systems, Proceedings of 20th International Conference on Distributed Computing Systems (ICDCS'2000) JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 26

Replication Requirements in Mobile Context Disconnected operations require Asynchronous Multi Master Replication More frequent reconciliation because of Lazy replication Message delays JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 27

Replication Techniques - Summary Replication classification scheme Update propagation Eager / synchronous primary server Lazy / asynchronous Update everywhere Lazy / asynchromnous Update location JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 28

Agenda Motivation Mobile Computing Synchronization techniques Replication techniques in databases Oracle Lite 10g – A Mobile Database Solution Peer to Peer Mobile Computing Future Applications and Developments JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 29

Oracle Lite 10g – A Mobile Database Solution Supports disconnected mobile database applications Local database (snapshot) on mobile device Content: subset of enterprise database Synchronization with enterprise database Central component: Mobile Server Application Model Publication Deployed on Mobile Server Distribution of application code Distribution of SQL defined snapshots (controllable by variable) JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 30

Oracle Lite 10g Architecture Source: Oracle® Database Lite Developer's Guide 10g (10.0.0) Part No. B13788-0 JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 31

Oracle Lite 10g Synchronization Process I Snapshot Keeps track of local modifications Three types of publication items (application depended) Complete refresh All data items Fast refresh Incremental (updates and inserts) Queue based refresh No refresh (e.g. data collection: newly created data items) Asynchronous synchronization Central component Mobile Server Via „In queues“ and „Out queues“ JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 32

Oracle Lite 10g Synchronization Process II Source: Oracle® Database Lite Developer's Guide 10g (10.0.0) Part No. B13788-0 JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 33

Oracle Lite 10g Evaluation Well suited for „build from scratch“ Enterprise database directly accessible from Mobile Server „Mobile only“ or „Designed for mobility“ applications Existing data access layers not supported (EJB, CORBA, other) Duplication of mission-critical data access logic Possible code portation efforts Maintenance efforts increase Benefits reduced or outweighted JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 34

Agenda Motivation Mobile Computing Synchronization techniques Replication techniques in databases Oracle Lite 10g – A Mobile Database Solution Peer to Peer Mobile Computing Future Applications and Developments JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 35

Peer to Peer Mobile Computing Shortcommings of client server model No true mobility No communication between mobile devices Requirements Any to any communication All devices equal Large replication factors Careful data replication JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 36

The Ward Model I Ward 2 Ward 3 Ward member Ward 1 Ward master Overlapped ward member JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 37

The Ward Model II Dynamically reconfigurable synchronization topology All units equal mobile peers: Intra ward mobility Inter ward mobility Ward changing (heavyweight) Ward overlapping (lightweight) Ward master: Re-electable Knows ward members No storage of ward member data Manages data import / export for ward JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 38

The Ward Model III Second, top-level ward: all ward master Ward set: data of all ward members within one ward Changes over time Optimistically replicated within ward Selective replication: intra- and inter-ward JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 39

Two Tier Replication Characteristics Comparable with ward model Static backbone of stationary replicated database servers Mobile replicas connect eventually at all stationary replicas Benefit Compromise between efficiency and true mobility Possible improvement: Combination with ward model (selective replication) JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 40

Agenda Motivation Mobile Computing Synchronization techniques Replication techniques in databases Oracle Lite 10g – A Mobile Database Solution Peer to Peer Mobile Computing Future Applications and Developments JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 41

Future Applications and Developments Car to car communication (http://www.car-to-car.org) Mobile network of communicating cars Security services Development of replication schemes, based on ward model ? Moving wards Multi level ward hierachy JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 42

Thanks for your attention ! JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 43

Discussion Try to describe other real world usage scenarios of mobile applications, where synchronization and replication techniques need to be used Which technologies could be helpful for application level data replication and integration. Think of technologies in the field of Enterprise Application Integration (EAI). Which advantages and disadvantages can be identified with respect to requirements in mobile applications JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 44

Appendix I: SyncML Example Source: SyncML Representation Protocol, version 1.0.1, http://www.syncml.org/docs/syncml_represent_v101_20010615.pdf JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 45

Appendix II: Majority Consensus Algorithm I Secondary 1 Secondary 1 Secondary 2 Network Partition Secondary 2 Primary Primary Secondary 3 Secondary 4 Secondary 4 Secondary 3 JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 46

Appendix II: Majority Consensus Algorithm II Secondary 2 Secondary 1 2,5 < 2 3 Inactive Partition Working partition Old Primary New Primary Secondary 4 JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 47

Appendix III: Quorum Consensus Algorithm I Weight: 1 Secondary 1 Weight: 1 Weight: 2 Secondary 1 Primary Primary Weight: 2 JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 48

Appendix III: Quorum Consensus Algorithm II > Partition weight = 2 Partition weight = 1 Working partition Weight: 2 Inactive partition Weight: 1 Primary Secondary 1 JASS 05 Synchronization and Replication in the Context of Mobile Applications © Alexander Stage 49