1. 2004 謝俊瑋 NTU, CSIE, CMLab 1 A Rule-Based Video Annotation System Andres Dorado, Janko Calic, and Ebroul Izquierdo, Senior Member, IEEE

2. 謝俊瑋 NTU, CSIE, CMLab 2 2004 Outline Introduction System overview Learning unit Automatic video annotation Result Conclusion

3. 謝俊瑋 NTU, CSIE, CMLab 3 2004 Introduction Semantic gap: information extracted from visual data and interpretation of the same data for a user in a given situation. Rules needed to infer a set of high-level concepts from low-level descriptors cannot be defined a priori. Knowledge embedded in the database and interaction with an expert user is exploited to enable system learning.

4. 謝俊瑋 NTU, CSIE, CMLab 4 2004 System overview System component  Learning unit Low-level feature extraction Knowledge representation Rule mining  Annotation unit Fuzzification Fuzzy inference Defuzzification

5. 謝俊瑋 NTU, CSIE, CMLab 5 2004 Learning unit

6. 謝俊瑋 NTU, CSIE, CMLab 6 2004 Low-level feature extraction Temporal features of video and its structure are considered as the foremost expressive elements One-dimensional frame-to-frame difference metrics[18][19] Shot change: local maxima of simplified curve Key frame: negative peaks in the second derivative of the simplified metric

7. 謝俊瑋 NTU, CSIE, CMLab 7 2004 Knowledge representation. Each word corresponds to a fuzzy set, which is the “interpretation” of w. Membership function:

8. 謝俊瑋 NTU, CSIE, CMLab 8 2004 Rule mining(1/3) Log of transactions T  T = [t 0, t 1, t 2, …, t n ]  t k = W f ∪ W c  W f are names of fuzzy sets mapped by M  W c is a set of words denoting concepts Association rule mining

9. 謝俊瑋 NTU, CSIE, CMLab 9 2004 Rule mining(2/3) Let I = {i 1, i 2, …, i n } be a set of literals called items. D be a set of transactions where each transaction T is a set of items such that T I The rule holds in the transaction set D with confidence c if c% of transactions in D containing X also contains Y. The rule holds in the transaction set D with support s if s% of transactions in D contains.

10. 謝俊瑋 NTU, CSIE, CMLab 10 2004 Rule mining(3/3)

11. 謝俊瑋 NTU, CSIE, CMLab 11 2004 Automatic video annotation Fuzzification  Map the input variable to the degree of membership of fuzzy sets  Same process as the knowledge representation unit

12. 謝俊瑋 NTU, CSIE, CMLab 12 2004 Fuzzy inference Use rules in the form:  IF THEN Condition expresses the instances of low- level features Actions denotes annotations Calculate fuzzy output values for the corresponding variable

13. 謝俊瑋 NTU, CSIE, CMLab 13 2004 Defuzzification Combine the fuzzy values of each output variable to obtain a real number for each variable Using weighted average method

14. 謝俊瑋 NTU, CSIE, CMLab 14 2004 Result(1/4) Over 100 randomly chosen MPEG2 video with duration of 2-6 minute

15. 謝俊瑋 NTU, CSIE, CMLab 15 2004 Result(2/4)

16. 謝俊瑋 NTU, CSIE, CMLab 16 2004 Result(3/4)

17. 謝俊瑋 NTU, CSIE, CMLab 17 2004 Result(4/4)

18. 謝俊瑋 NTU, CSIE, CMLab 18 2004 Conclusion Extensions:  Easily extended to support audio  Scenes or groups of videos  Evaluation on few focused features Disadvantages:  Too many keywords can cause conflict Future works:  Utilization of user relevance feedback

19. 謝俊瑋 NTU, CSIE, CMLab 19 2004 Reference [18] L.J. Latecki and R. Lakimper, “Convexity rule for shape decomposition based on discrete contour evolution,” in Computer Vision and Image Understanding. New York: Academic, 1999 [19] J. Calic and E. lzquierdo, “A multiresolution technique for video indexing and retrieval,” in Proc. IEEE ICIP2002 [20] T.J. Ross, Fuzzy Logic With Engineering Applications. New York: McGraw-Hill, 1995.