2. 1 Noget helt andet… Platon vil gerne være vært (i Århus) for et BIT møde i efteråret – SOA eller MDM – Fint for mig, men hvad siger i ? Platon inviterer alle til www.bi2006.dkwww.bi2006.dk – 7-8 juni – Special pris for BIT medlemmer: 2995 kr. – Tilmelding via Jørgen Davidsen, jda@platon.net

3. Lineage Tracing in Data Warehouses Torben Bach Pedersen Based on work by Yingwei Cui and Jennifer Widom Stanford University Database Group

4. 3 Motivation: Data Warehousing Data Warehouse Source 1Source 2Source 3 Lucrative Fields Databases $8800K Theory $320K Networks $800K StudentsEnrollmentsCourses Wow?! Databases $8800K

5. 4 CoursesEnrollmentsStudents Oh, I see... Source 1Source 2Source 3 Lineage Tracer Data Warehouse Lucrative Fields Database 1800 Theory $320K Networks $800K Databases $8800K CS145 Ted CS154 Joe CS244 Bob CS145 Ann CS245 Jane …… Bob MS $1K Jane Web $5K Ann BS $1K Joe BS $1K Ted Web $5K ……… CS145 Databases CS154 Theory CS244 Networks CS245 Databases

6. 5 The Data Lineage Problem Data warehouses integrate data from multiple sources for analysis and mining Data lineage Data lineage: given data item o in the warehouse, which data items in the sources were used to derive o? Sometimes called “drill-through” in industry – “Drill-through” often limited

7. 6 Challenges Warehouse of relational views over relational sources – What is a good formal definition for lineage? – How do we trace data lineage for arbitrary views? – How do we make it efficient? Warehouse defined by graph of data transformations – No fixed, well-defined relational operators – Large transformation sequences and graphs

8. 7 Outline of Talk Part 1: Lineage tracing for relational views Part 2: Lineage tracing for general data transformations

9. 8 Part 1: Lineage Tracing for Relational Views Declarative definition of data lineage Lineage tracing algorithms Using auxiliary views for efficient lineage tracing Experimental results (small sample)

10. 9 Views We Consider Relational algebra Arbitrary use of aggregation Set semantics Also in thesis – Set operators – Bag semantics        RST V

11. 10  V V =  (  ( R S )) Y,sum(Z)X >Z R S X Y Z 3 2a b b 8 8 0 6 Y sum a2 b6 X Y Z 32a 80 89 86 b b b X Y 3 a Y Z 2 a 0b 9b 6b 8b Y,sum(Z)X >Z TU b6 b80 b86 80 86 b b 0b 6b 8b Simple Lineage Example select Y,sum(Z) from R natural join S where X>Z group by Y

12. 11 Lineage for Relational Operators Unary relational operators definition took a long time op R R*t Lineage of t according to op is the maximal subset R*  R such that (1) op(R*) = {t} - output of R* through op is t (2)  t*  R*: op({t*})  - op used on t* is nonempty 

13. 12 Example 1 – the two conditions ensure that only tuples contributing to t are included in lineage R  X Y Z 3 2a b b 8 8 0 6 32a 80 89 86 b b b X >Z Lineage of t according to op is the maximal subset R*  R such that (1) op(R*) = {t} (2)  t*  R*: op({t*})  Lineage for Relational Operators b86 86b

14. 13 Example 2 –”maximal” requirement ensures that (8,b,0) tuple in included in (b,6) lineage R  X Y Z 3 2a b b 8 8 0 6 Y sum a2 b6 Y,sum(Z) maximal Lineage of t according to op is the maximal subset R*  R such that (1) op(R*) = {t} (2)  t*  R*: op({t*})  Lineage for Relational Operators b6 b80 b86

15. 14 N-ary relational operators ( , ,  ) – lineage unique Lineage for Relational Operators maximal Lineage of t according to op is the maximal subsets R i *  R i for i = 1..n such that (1) op(R 1 *, …, R n *) = {t} (2)  t i *  R i *: op(R 1, …, {t i *}, …, R n )  op R1R1 * * R2R2 R2R2 R1R1

16. 15 Lineage for Relational Views Lineage of a tuple set is union of lineage of each tuple in the set Lineage for views is defined recursively => naive, but inefficient, algorithm (need to recompute/store all intermediate results) op 1 2 VU R1R1 R2R2 t U* * * R1R1 R2R2 Lineage of t is  R 1 *, R 2 * 

17. 16 Lineage Tracing segmented normal form (SPJ+agg) Convert view into segmented normal form (SPJ+agg)    E 1 … E n  Each segment tracing query Generate one tracing query for each segment Apply tracing queries recursively – # non-top  + 1 Proof: lineage result is unaffected by normalization and segment-level tracing Proof: lineage result is unaffected by normalization and segment-level tracing

18. 17 Tracing Query for One Segment VY sum a2 b6 R S TQ = Split (  ( R S )) X >Z  Y=b R,S  Y,sum(Z)  X >Z b 6 b X Y 3 a 8 Y Z 2 a 0 9b b  R*={(8,b)}, S*={(b,0),(b,6)}  b 0 6 b b 8 b6 V =  (  ( R S )) X >ZY,sum(Z) Split = ”unjoin” – project over R+S schemas

19. 18 Recursive Tracing Procedure   VW avg p 4 q 6 U R S X Y 3 a Y Z 2 a 0b 9b 6b 8b T Y sum a2 b 6 Y W ap p q b b  TQ = Split (  ( U T )) W=q1U,T TQ = Split (  ( R S )) X >Z  Y=b 2R,S  b 6 qb 8b 0b 6b q 6  R*={(8,b)}, S*={(b,0),(b,6)}, T*={(b,q)}  8b 0b 6b qb V =  (  (  ( R S )) T )) W, avg(sum)Y,sum(Z)X >Z

20. 19 Making It Efficient Source accesses are usually expensive or impossible Need some intermediate results for lineage tracing auxiliary views  Store auxiliary views at the warehouse – Reduce or eliminate source accesses – Reduce recomputation of intermediate results

21. 20 Aux View Example

22. 21 Aux View Example

23. 22 Auxiliary Views There are many possible auxiliary views For single-segment views – Identified 10 possible auxiliary view schemes – Studied performance tradeoffs For arbitrary views – Hard optimization problem – Exhaustive and heuristic algorithms – Performance study    R 1 … R n 

24. 23 Single Segment Schemes Store nothing (NO) Store Base Tables (BT) Store Lineage Views (LV) Store Split Lineage Tables (SLT) Store Partial Base Tables (PBT) Store Base Table Projections (BP) Store Lineage View Projections (LP) Self-maintainable variations: LV-S, SLT-S, PBT-S

25. 24 + Always improve lineage tracing – Must be maintained when sources change + Can also help with maintenance of original user views Auxiliary Views: Performance Tradeoffs

26. 25 Auxiliary View Schemes for Single-Segment Views Parameters: - 3-way SPJ view - sources: 10MB each - disk: 1Mbps - network: 50kbps - 1000 operations - q/u ratio = 4 Measurements: - tracing time - maintenance time

27. 26 Auxiliary View Selection Algorithms for Arbitrary Views

28. 27 Part 2: Transformation Graphs Lineage definition Tracing algorithms Combining transformations for lineage tracing Experimental results (tiny sample) Source 1 Data Warehouse Source 2 Source 3 T6T6 T4T4 T5T5 T3T3 T2T2 T1T1

29. 28 T1T1 T3T3 T4T4 T6T6 T7T7 T5T5 id cust date prod-list 1 A 2/8/99 1(10),2(10) 2 C 4/5/99 2(5),3(10) 3 D 6/1/99 1(20),2(10) 4 B 8/6/99 1(10),3(5) 5 D 10/8/99 1(5),3(10) 6 B 12/1/99 2(10),3(10) id name price valid 1 imac 1200 10/1/98- 2 vaio 2400 6/1/98-9/1/99 2 vaio 1800 9/2/99- 3 palm 500 2/1/98-7/1/98 3 palm 400 7/2/98-9/1/99 3 palm 300 9/2/99- name avg3 Q4 palm 2K 6K 3 palm 400 7/2/98-9/1/99 3 palm 300 9/2/99- 2 C 4/5/99 2(5),3(10) 4 B 8/6/99 1(10),3(5) 5 D 10/8/99 1(5),3(10) 6 B 12/1/99 2(10),3(10) SalesJump Order Product T2T2 Transformation Example selection “join”splitpivotprojectionselectionprojection

30. 29 Lineage for General Transformations transformation A transformation can be an arbitrary program T  select … from … where …  main(int argc, char** argv) {…}  sed “s/string1/string2/g” … ? – One extreme: relational operators – Another extreme: we know nothing about T – Middle ground: based on transformation properties

31. 30 Transformation Properties Transformation classes Additional properties – Transformation subclasses – Schema information – Provided inverse or tracing procedure

32. 31 i  I  I: T(I) =  T({i}) dispatcher T*(o) = {i | o  T({i})} Transformation Classes Produces 0 or more output items per input item Applying T on complete set is the same as on each input item separately

33. 32 Dispatcher Example id cust date prod-list 1 A 2/8/99 1(10),2(10) 2 C 4/5/99 2(5),3(10) 3 D 6/1/99 1(20),2(10) 4 B 8/6/99 1(10),3(5) 5 D 10/8/99 1(5),3(10) 6 B 12/1/99 2(10),3(10) Order id cust date pid quant 1 A 2/8/99 1 10 1 A 2/8/99 2 10 : : : 5 D 10/8/99 1 5 5 D 10/8/99 3 10 6 B 12/1/99 2 10 6 B 12/1/99 3 10 T1T1 O1O1 5 D 10/8/99 1(5),3(10) 5 D 10/8/99 1 5 5 D 10/8/99 3 10 5 D 10/8/99 1(5),3(10) A non-relational operator, but a typical dispatcher

34. 33 i  I  I: T(I) =  T({i}) dispatcher  I and T(I)={o 1 …o n }:  unique partition I 1..I n of I s.t. T(I k ) = {o k } aggregator T*(o k ) = I k T*(o) = {i | o  T({i})} Transformation Classes

35. 34 Aggregator Example T4T4 name Q1 Q2 Q3 Q4 imac 12K 24K 12K 6K vaio 24K 12K 24K 18K palm 0K 4K 2K 6K O3O3 O4O4 oid name date price quant 1 imac 2/8/99 1200 10 1 vaio 2/8/99 2400 10 2 vaio 4/5/99 2400 5 3 imac 6/1/99 1200 20 3 vaio 6/1/99 2400 10 4 imac 8/6/99 1200 10 4 palm 8/6/99 400 5 5 imac 10/8/99 1200 5 5 palm 10/8/99 300 10 6 vaio 12/1/99 1800 10 6 palm 12/1/99 300 10 2 palm 4/5/99 400 10 4 palm 8/6/99 400 5 6 palm 12/1/99 300 10 palm 0K 4K 2K 6K 5 palm 10/8/99 300 10 palm 0K 4K 2K 6K 2 palm 4/5/99 400 10 4 palm 8/6/99 400 5 6 palm 12/1/99 300 10 5 palm 10/8/99 300 10 T4 computes quarterly sales per product by ”pivoting” Again, a non-relational operator, but a typical aggregator

36. 35 i  I  I: T(I) =  T({i}) dispatcher  I and T(I)={o 1 …o n }:  unique partition I 1..I n of I s.t. T(I k ) = {o k } aggregatorblack-box All others T*(o k ) = I k T*(o) = I T*(o) = {i | o  T({i})} Transformation Classes

37. 36 Most transformations are dispatchers, aggregators, or their compositions A transformation can be both dispatcher and aggregator – Proof: Lineage definitions are then equivalent Transformations can be relational operators – Lineage definitions same as relational definitions Transformation Classes

38. 37 Transformation Properties Transformation classes Additional properties – Transformation subclasses – Schema information – Provided inverse or tracing procedure

39. 38 Transformation Subclasses Permit more efficient lineage tracing Filter is a special dispatcher – Each input data item produces itself or nothing Context-free aggregator – Whether two input data items are in the same partition is independent of other items Key-preserving aggregator – Any subset of an input partition always produces the same output key

40. 39 Tracing Example: Aggregators Consider T(I) = {o 1 …o n } Tracing the lineage of o for aggregator – Partition input I into I 1 …I n such that T(I k ) = {o k } – Return I k such that T(I k ) = {o} Tracing the lineage of o for context-free aggregator – Partition input I into I 1 …I n such that |T(I k )| = 1 – Return I k such that T(I k ) = {o} – 2^n versus n^2 running time !

41. 40 Schema Information Input schema A=(A 1 …A n ) and key A key Output schema B=(B 1 …B n ) and key B key Schema mappings: f(A)  B and A  g(B) Transformations with special schema mappings – Forward key-map: f(A)  B key – Backward key-map: A key  g(B) – Backward total-map: A  g(B) – More efficient tracing for these

42. 41 Tracing Example: Forward Key-Maps T4T4 O3O3 O4O4 name Q1 Q2 Q3 Q4 imac 12K 24K 12K 6K vaio 24K 12K 24K 18K palm 0K 4K 2K 6K oid name date price quant 1 imac 2/8/99 1200 10 1 vaio 2/8/99 2400 10 2 vaio 4/5/99 2400 5 3 imac 6/1/99 1200 20 3 vaio 6/1/99 2400 10 4 imac 8/6/99 1200 10 4 palm 8/6/99 400 5 5 imac 10/8/99 1200 5 5 palm 10/8/99 300 10 6 vaio 12/1/99 1800 10 6 palm 12/1/99 300 10 2 palm 4/5/99 400 10 4 palm 8/6/99 400 5 6 palm 12/1/99 300 10 5 palm 10/8/99 300 10 ”name” is carried over as key - trace of ”palm” is easy : the O3 tuples with name = ’palm’

43. 42 Other Properties Transformation author provides Tracing Procedure Provided Transformation Inverse T –1 – If T is an aggregator, then o’s lineage is T –1 ({o}) – Not always true for dispatchers or black-boxes

44. 43 Tracing Procedures PropertyProcedure# T Calls# Accesses dispatcher TraceDS O(|I|) aggregator TraceAG O(2 |I| ) black-box return I; 0O(|I|) filter return o; 00 context-free aggr. TraceCF O(|I| 2 ) key-preserving aggr. TraceKP O(|I|) forward key-map TraceFM 0O(|I|) backward key-map TraceBM 0O(|I|) backward total-map TraceTM 00 Provided tracing-proc.provided??

45. 44 Property Hierarchy ANY provided tracing-proc. or inverse black-box aggregator dispatcher context-free aggr. key-preserving aggr. filter forward key-map backward key-map total-map

46. 45 Summary of Our Approach for One Transformation Properties are provided with transformations – Specified by the transformation author – Declared in prepackaged transformations – Derived using recent techniques [Clio01, RB01] The best property of a transformation is selected based on the hierarchy The tracing procedure using the best property is called at tracing time Indexing techniques

47. 46 Transformation Sequences Naive algorithm traces backwards one transformation at a time – Need all intermediate results –Poor performance for long sequences T1T1 T2T2 T3T3 TnTn I O

48. 47 T1T1 T2T2 T3T3 TnTn I O T’TnTn I O Combine transformations and trace as one – Reduces number of intermediate results – By combining judiciously  Reduces tracing cost  Doesn’t lose accuracy Transformation Sequences