1. IBM QBIC: Query by Image and Video Content Jianping Fan Department of Computer Science University of North Carolina at Charlotte Charlotte, NC 28223 http://www.cs.uncc.edu/~jfan

2. How can I access video in database over networks? 1. How to access video ? 2. How to represent video ? 3. How to index large-scale videos ? 4. How to access videos in database ? 5. How to transmit query results over IP ? Networks 6. How to control user’s access ?

3. 1. System Architecture How to access video in database? ImagesVideos Shot extraction 1. Representative Frame 2. Motion-based objects 3. shots Object Identification R-frames scene sketch Positional color/texture object Location/color User-defined Color/texture video Object motion Camera motion Feature extraction sceneobjectsshots Motion objects

4. 1. System Architecture How to access video in database? user Query interface Color Texture Shape Multi-object Sketch Location Text Positional color/texture object motion camera motion user defined existing image Match engine Database Indexing Returned via similarity order

5. 2. How to access image or video in QBIC? a. Video shots b. Video objects or sketches & drawings c. Representative frames Access approaches a. Example images b. user-constructed sketches or drawings c. Selected color and texture patterns d. Camera & object motion e. Other graphical information d. Motion types e. Other information

6. 2. How to obtain accessing units in QBIC? Shot Detection

7. 2. How to obtain accessing units in QBIC? a. Difference Calculations b. Automatic Decision MakingVia Pre-Defined Thresholds.

8. How we do the shot detection?

9. 2. How to obtain accessing units in QBIC? Object extraction

10. 2. How to obtain accessing units in QBIC? Object extraction

11. How we can do the object extraction?

12. 3. How to represent these video units? Images Global color Global texture Positional color Positional texture Sketch, shape User-defined color/texture

13. 3. How to represent these video units? Videos Global color Global texture Positional color Positional texture Sketch, shape User-defined color/texture Camera motion/object motion

14. Color HSV color histogram, dominant color, … Texture Edge histogram, wavelet coefficients, Tamura features, … Motion Directional motion histogram, Camera motion, … Other features Video Sequence Shot 1 Shot i Shot n How we do the shot representation?

15. 4. How to index images/videos in QBIC?........................ feature space Videos in Database

16. 4. How to index images/videos in QBIC? High-dimensional visual features K-L Transform to reduce dimensions Low-dimensional R*-tree indexing

17. 4. How to index images/videos in QBIC? Overlap on two Dimensions!

18. 4. How to index images/videos in QBIC? Karhunen-Loeve Transformation New Eigenvectors M is the matrix for videos! S is the KL transform matrix!

19. 5. How to realize query in QBIC?

22. How we can do the video query?

23. 5. How to realize query in QBIC?

24. How we can do the mosaic?

25. Why we use mosaic for video representation?

26. 6. What lost by QBIC? a. Mapping from visual features to semantic concepts It is hard, but we have to do this. Why?.......... Visual Features...... Semantic Clusters................. Video Contents in Database Weighted mapping? How to do this mapping?

27. 6. What lost by QBIC? b. High-dimensional visual indexing It is a basic problem in database area, but only database people cannot solve this challenging problem for visual indexing! Video in Database Cluster 1Cluster iCluster n Subcluster 11Subcluster 1j Subcluster n1 Subcluster nl Subregion 111 Subregion 11k Subregion nl1Subregion nlm object1111 object nlm1 Disk for Cluster 1 Disk for Cluster i Disk for Cluster n

28. 6. What lost by QBIC? c. User input in the query procedure: QBIC can permit user to select something.

29. 6. What lost by QBIC? d. How to integrate keywords with visual features?

30. 6. What lost by QBIC? e. How to provide user-intensive browsing?

31. 7. What happen now on QBIC? You can find the current version of QBIC system on: http://wwwqbic.almaden.ibm.com/ Homework: What kind of technique we have discussed used in QBIC?

32. 8. Other Projects a.Chabot at UC Berkeley http://www.cs.berkeley.edu/~ginger/chabot.html b. Viper at Europe http://viper.unige.ch/ c. Virage http://www.virage.com/

33. 8. Other Projects d. PicHunter at NEC http://www.neci.nj.nec.com/homepages/vision/index.html e. Ifind at Microsoft http://wwww.microsoft.com/china/research/group/ f. Photobook at MIT http://www-white.media.mit.edu/~tpminka/photobook/