DataWarp: Making Progress Despite Inconsistent Data Stephen Crouch Peter Henderson Robert John Walters School of Electronics and Computer Science, University of Southampton, UK

Outline Background Traditional Philosophy DataWarp Example Conclusion

Modern Systems: No longer exist in private environments Are connected to each other Use data which Is (at least partially) replicated Can be out of date Contains errors They don’t own

Traditional approach “Everything must have a correct value” We must drive out the imperfections Implement systems to make sure data remains consistent Don’t do anything unless sure it is right

Examples Transactions Elaborate schemes which ensure data remains consistent Compensations Less elaborate and restrictive Relax some restrictions of transactions but expose intermediate states

Single Datum World Transactional systems never leave left-most column Compensation systems can, but Temporarily Make sure they know how to get back

But we can never achieve full consistency “Inconsistencies” which are deliberate Different notions of consistency Ownership Cost The accumulated body of data is too big

DataWarp, an alternative We can’t “fix” the data so: We have to “fix” the applications DataWarp Can’t give up when inconsistency found Do the best you can with what you have Be prepared to make corrections

Single Datum World DataWarp: Accepts being in leftmost column is unlikely

Grid Scheduling Example Classical approach to any workflow Find and execute the first task Wait for it to complete Execute the next task … Works, but time wasted waiting

Example Workflow as Text Data DI # Input Data DJK # Output J or K Data DA,DB,DC,DH # Other Output Job A,B,C,H,J,K # Tasks A.submitJob(DI) A.waitFor() DA = A.getResults() parallel { B.submitJob(DA) B.waitFor() DB = B.getResults() } and { H.submitJob(DA) DH = H.getResults() if ( some_predicate(DH) ) { J.submitJob(DH) J.waitFor() DJK = J.getResults() } else { K.submitJob(DH) K.waitFor() DJK = K.getResults() } C.submitJob(DB, DJK) C.waitFor() DC = C.getResults()

Example Workflow as Diagram

Notice Both B and H can start as soon as A completes and can run at the same time Whether we do J or K depends on result of H C needs output from B and J or K Processing time for each job includes waiting in the queue

Execution times:

Optimisations 1 Anticipation Put jobs in the queue so they come to the head of the queue just as we have the data to execute them Run more than one job at a time Users do this manually Jobs put in slow moving queues ready for when needed

The Schedule ProcessExecution TimeDelay for placeholder job A70 B207 C4327 H57 J1112 K8

Features Start B, H together Sequentially C finishes at 116 By running B in parallel with H,J,K this improves to 88 Anticipating need for jobs this is improved to 76

Optimisations 2 Suppose queue prediction is too pessimistic Jobs for J,K arrive at head of queue while H still working Start both Abandon one when H completes Suppose H fails/still working when B finishes and C ready Pick output from one of J,K Complete the workflow

Conclusion Applications have to manage in connected environment Insisting on complete, consistent data is no longer acceptable DataWarp applications can live with uncertain data They continue where others fail