Presentation is loading. Please wait.

Presentation is loading. Please wait.

CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #14: Text – Part I C. Faloutsos.

Published byLuc Fairhurst Modified over 10 years ago

Similar presentations

Presentation on theme: "CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #14: Text – Part I C. Faloutsos."— Presentation transcript:

1 CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #14: Text – Part I C. Faloutsos

2 CMU SCS Copyright: C. Faloutsos (2012) Must-read Material MM Textbook, Chapter 6 15-8262

3 CMU SCS Copyright: C. Faloutsos (2012) Outline Goal: ‘Find similar / interesting things’ Intro to DB Indexing - similarity search Data Mining 15-8263

4 CMU SCS Copyright: C. Faloutsos (2012) Indexing - Detailed outline primary key indexing secondary key / multi-key indexing spatial access methods fractals text multimedia... 15-8264

5 CMU SCS Copyright: C. Faloutsos (2012) Text - Detailed outline text –problem –full text scanning –inversion –signature files –clustering –information filtering and LSI 15-8265

6 CMU SCS Copyright: C. Faloutsos (2012) Problem - Motivation Eg., find documents containing “data”, “retrieval” Applications: 15-8266

7 CMU SCS Copyright: C. Faloutsos (2012) Problem - Motivation Eg., find documents containing “data”, “retrieval” Applications: –Web –law + patent offices –digital libraries –information filtering 15-8267

8 CMU SCS Copyright: C. Faloutsos (2012) Problem - Motivation Types of queries: –boolean (‘data’ AND ‘retrieval’ AND NOT...) 15-8268

9 CMU SCS Copyright: C. Faloutsos (2012) Problem - Motivation Types of queries: –boolean (‘data’ AND ‘retrieval’ AND NOT...) –additional features (‘data’ ADJACENT ‘retrieval’) –keyword queries (‘data’, ‘retrieval’) How to search a large collection of documents? 15-8269

10 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning Build a FSA; scan c a t 15-82610

11 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning for single term: –(naive: O(N*M)) ABRACADABRAtext CAB pattern 15-82611

12 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning for single term: –(naive: O(N*M)) –Knuth Morris and Pratt (‘77) build a small FSA; visit every text letter once only, by carefully shifting more than one step ABRACADABRAtext CAB pattern 15-82612

13 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning ABRACADABRAtext CAB pattern CAB... 15-82613

14 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning for single term: –(naive: O(N*M)) –Knuth Morris and Pratt (‘77) –Boyer and Moore (‘77) preprocess pattern; start from right to left & skip! ABRACADABRAtext CAB pattern 15-82614

15 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning ABRACADABRAtext CAB pattern CAB 15-82615

16 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning ABRACADABRAtext OMINOUS pattern OMINOUS Boyer+Moore: fastest, in practice Sunday (‘90): some improvements 15-82616

17 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning For multiple terms (w/o “don’t care” characters): Aho+Corasic (‘75) –again, build a simplified FSA in O(M) time Probabilistic algorithms: ‘fingerprints’ (Karp + Rabin ‘87) approximate match: ‘agrep’ [Wu+Manber, Baeza-Yates+, ‘92] 15-82617

18 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning Approximate matching - string editing distance: d( ‘survey’, ‘surgery’) = 2 = min # of insertions, deletions, substitutions to transform the first string into the second SURVEY SURGERY 15-82618

19 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning string editing distance - how to compute? A: 15-82619

20 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning string editing distance - how to compute? A: dynamic programming cost( i, j ) = cost to match prefix of length i of first string s with prefix of length j of second string t 15-82620

21 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning if s[i] = t[j] then cost( i, j ) = cost(i-1, j-1) else cost(i, j ) = min ( 1 + cost(i, j-1) // deletion 1 + cost(i-1, j-1) // substitution 1 + cost(i-1, j) // insertion ) 15-82621

22 CMU SCS Copyright: C. Faloutsos (2012) String editing distance  SURVEY  0123456 S1 U2 R3 G4 E5 R6 Y7 15-82622

23 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1 U2 R3 G4 E5 R6 Y7 15-82623

24 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S10 U2 R3 G4 E5 R6 Y7 15-82624

25 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U210 R32 G43 E54 R65 Y76 15-82625

26 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R321 G432 E543 R654 Y765 15-82626

27 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R3210123 G4321 E5432 R6543 Y7654 15-82627

28 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R3210123 G4321123 E54322 R65433 Y76544 15-82628

29 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R3210123 G4321123 E5432212 R654332 Y765443 15-82629

30 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R3210123 G4321123 E5432212 R6543322 Y765443 15-82630

31 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R3210123 G4321123 E5432212 R6543322 Y7654432 15-82631

32 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R3210123 G4321123 E5432212 R6543322 Y7654432 15-82632

33 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R3210123 G4321123 E5432212 R6543322 Y7654432 15-82633

34 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R3210123 G4321123 E5432212 R6543322 Y7654432 15-82634

35 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R3210123 G4321123 E5432212 R6543322 Y7654432 15-82635

36 CMU SCS Copyright: C. Faloutsos (2012) String editing distance SURVEY 0123456 S1012345 U2101234 R3210123 G4321123 E5432212 R6543322 Y7654432 subst. del. 15-82636

37 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning Complexity: O(M*N) (when using a matrix to ‘memoize’ partial results) 15-82637

38 CMU SCS Copyright: C. Faloutsos (2012) Full-text scanning Conclusions: Full text scanning needs no space overhead, but is slow for large datasets 15-82638

Download ppt "CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #14: Text – Part I C. Faloutsos."

Similar presentations

CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #17: Text - part IV (LSI) C. Faloutsos.

CMU SCS : Multimedia Databases and Data Mining Lecture #17: Text - part IV (LSI) C. Faloutsos.
Chapter 7 Space and Time Tradeoffs Copyright © 2007 Pearson Addison-Wesley. All rights reserved.

Chapter 7 Space and Time Tradeoffs Copyright © 2007 Pearson Addison-Wesley. All rights reserved.
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #7: Spatial Access Methods - IV Grid files, dim. curse C. Faloutsos.

CMU SCS : Multimedia Databases and Data Mining Lecture #7: Spatial Access Methods - IV Grid files, dim. curse C. Faloutsos.
Indexing DNA Sequences Using q-Grams

Indexing DNA Sequences Using q-Grams
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #4: Multi-key and Spatial Access Methods - I C. Faloutsos.

CMU SCS : Multimedia Databases and Data Mining Lecture #4: Multi-key and Spatial Access Methods - I C. Faloutsos.
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture#1: Introduction Christos Faloutsos CMU www.cs.cmu.edu/~christos.

CMU SCS : Multimedia Databases and Data Mining Lecture#1: Introduction Christos Faloutsos CMU
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #19: SVD - part II (case studies) C. Faloutsos.

CMU SCS : Multimedia Databases and Data Mining Lecture #19: SVD - part II (case studies) C. Faloutsos.
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #11: Fractals: M-trees and dim. curse (case studies – Part II) C. Faloutsos.

CMU SCS : Multimedia Databases and Data Mining Lecture #11: Fractals: M-trees and dim. curse (case studies – Part II) C. Faloutsos.
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #7: Spatial Access Methods - Metric trees C. Faloutsos.

CMU SCS : Multimedia Databases and Data Mining Lecture #7: Spatial Access Methods - Metric trees C. Faloutsos.
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #10: Fractals - case studies - I C. Faloutsos.

CMU SCS : Multimedia Databases and Data Mining Lecture #10: Fractals - case studies - I C. Faloutsos.
Two-dimensional pattern matching M.G.W.H. van de Rijdt 23 August 2005.

Two-dimensional pattern matching M.G.W.H. van de Rijdt 23 August 2005.
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture#5: Multi-key and Spatial Access Methods - II C. Faloutsos.

CMU SCS : Multimedia Databases and Data Mining Lecture#5: Multi-key and Spatial Access Methods - II C. Faloutsos.
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture#2: Primary key indexing – B-trees Christos Faloutsos - CMU www.cs.cmu.edu/~christos.

CMU SCS : Multimedia Databases and Data Mining Lecture#2: Primary key indexing – B-trees Christos Faloutsos - CMU
Space-for-Time Tradeoffs

Space-for-Time Tradeoffs
String Searching Algorithm

String Searching Algorithm
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture#3: Primary key indexing – hashing C. Faloutsos.

CMU SCS : Multimedia Databases and Data Mining Lecture#3: Primary key indexing – hashing C. Faloutsos.
Temple University – CIS Dept. CIS616– Principles of Data Management V. Megalooikonomou Text Databases (some slides are based on notes by C. Faloutsos)

Temple University – CIS Dept. CIS616– Principles of Data Management V. Megalooikonomou Text Databases (some slides are based on notes by C. Faloutsos)
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture #11: Fractals: M-trees and dim. curse (case studies – Part II) C. Faloutsos.

CMU SCS : Multimedia Databases and Data Mining Lecture #11: Fractals: M-trees and dim. curse (case studies – Part II) C. Faloutsos.
15-826: Multimedia Databases and Data Mining

15-826: Multimedia Databases and Data Mining
CMU SCS 15-826: Multimedia Databases and Data Mining Lecture#1: Introduction Christos Faloutsos CMU www.cs.cmu.edu/~christos.

CMU SCS : Multimedia Databases and Data Mining Lecture#1: Introduction Christos Faloutsos CMU

Similar presentations

Ads by Google