1. Multimedia Analytics Jianping Fan Department of Computer Science University of North Carolina at Charlotte

2. Presentation Outlines What is Multimedia Analytics in my mind? What multimedia analytics can do for Multimedia Computing? What we (UNCC team) have done so far?

3. What is Multimedia Analytics? Multimedia analytics is science of multimedia computing facilitated by visual interface for interactive user’s inputs & assessments! 1. Multimedia Rept. 2. Computing Hypos 3. Decision Function 4. Computing Results Visual Analytics Semantic Gap Machine Side Human Side Assess & Inputs multimedia computing 1. Multimedia Rept. 2. Computing Hypos 3. Decision Function 4. Computing Results

4. What is Multimedia Analytics? Multimedia Analytics a. Machine-Based Multimedia Computing; b. Visualization of Multimedia Data, Knowledge, Hypotheses, Decision Functions & Results; c. Human-Computer Interaction for assessing & changing hypotheses for multimedia computing Loop

5. What Multimedia Analytics can do for Multimedia Computing? Multimedia Analytics Multimedia Computing Knowledge & Hypotheses Visualization User-System Interaction & Assessment Decision making Decision adjustment Hypothesis visualization Hypothesis assessment sampling & projection Hypothesis updating

6. What Multimedia Analytics can do for Multimedia Computing? Hypothesis & Decision Function Visualization and Assessment Computing Results & Knowledge Visualization, Exploration and Assessment Leveraging the advantages of both human beings on creative thinking and computers on large memory and computing capacity.

7. What we have done so far a. Interactive Similarity Function Assessment

8. What we have done so far a. Interactive Similarity Function Assessment

9. What we have done so far a. Interactive Similarity Function Assessment

10. What we have done so far a. Interactive Similarity Function Assessment

11. What we have done so far a. Interactive Similarity Function Assessment Inter-cluster visual correlation visualization Scalability

12. Feature Quality Evaluation

13. What we have done so far a. Interactive Similarity Function Assessment Similarity function is suitable for measuring visual similarity contexts? Similarity function combination (kernel combination) is good? HD projection can precisely preserve original visual similarity contexts? Feature quality is good?

14. What we have done so far Misleading Effects: Data Uncertainty Junk Images

15. What we have done so far Duplicates/Near-Duplicates Misleading Effects: Data Uncertainty

16. What we have done so far Multiple text terms may share similar semantic meaning! One single text term may have multiple semantic meanings! Misleading Effects: Data Uncertainty

17. What we have done so far Multi-Modal Information Association Misleading Effects: Data Uncertainty

18. What we have done so far Misleading Effects: Data Uncertainty

19. What we have done so far b. Interactive Decision Function Evaluation SVM Decision Boundary Visualization & Assessment

20. What we have done so far b. Interactive Decision Function Evaluation Concept 1Concept 2 GMM Model Visualization

21. What we have done so far b. Interactive Decision Function Evaluation

22. Interactive User Involvements via labeling

23. What we have done so far Updating kernel combinations (kernel weights); Updating projection optimization criteria to preserve similarity better! Updating decision function: margin between positive samples and negative samples! Updating hypotheses for data representation & similarity characterization! b. Interactive Decision Function Evaluation

24. Enlarge the margin between two classes!

25. Larger margin has good generalization property!

26. What we have done so far c. Interactive Knowledge Exploration

27. What we have done so far c. Interactive Knowledge Exploration

28. What we have done so far c. Interactive Knowledge Exploration

29. What we have done so far c. Interactive Knowledge Exploration

30. What we have done so far d. Collaborative Multimedia Analytics People have strong motivations (self or social) to perform collaborate multimedia understanding!

31. What we have done so far d. Collaborative Multimedia Analytics

32. What we have done so far d. Collaborative Multimedia Analytics Learning task and training group organization; Human-computer communication on data and knowledge representation; Human-human communication on hypotheses and knowledge.

33. More information is available at: http://www.cs.uncc.edu/~jfan Q & A!