1. A Generic Virtual Content Insertion System Based on Visual Attention Analysis H. Liu 1, 2, S. Jiang 1, Q. Huang 1, 2, C. Xu 2, 3 1 Institute of Computing Technology, Chinese Academy of Sciences 2 China-Singapore Institute of Digital Media 3 National Lab of Pattern Recognition, Institute of Automation

2. Outline Motivation Related work The proposed Virtual Content Insertion (VCI) system Experimental results Conclusions 2015-9-18http://www.jdl.ac.cn2

3. Virtual Content Insertion 2015-9-18http://www.jdl.ac.cn3 Convenient Changeable Cost less To construct a generic VCI system

4. Challenge Advertisement insertion VS. Augmentation Software based VS. Hardware based Challenge Insertion time Insertion place Insertion method Insertion content

5. Related work– Insertion Time Insert advertisements into video prologue Be neglected Insert the ad into interesting segments Our method Temporal attention Higher Attentive Shots K. Wan, C. Xu, “Automatic Content Placement in Sports Highlights”, ICME, 2006. 2015-9-18http://www.jdl.ac.cn5

6. Related work– Insertion Place Static region Color consistent region Visual relevance measure Lower informative region Our method Spatial attention Lower Attentive Region C. Xu, etc., “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, 2004. K. Wan, etc., “Automatic Content Placement in Sports Highlights”, ICME, 2006. Y. Li, etc., “Real Time Advertisement Insertion in Baseball Video Based on Advertisement Effect”, ACM Multimedia, 2005. 2015-9-18http://www.jdl.ac.cn6

7. Related work – Insertion Method Challenge Camera parameters unknown Existing methods Structure of the scene Predefined landmarks Our method Affine transformation Global Motion Estimation X. Yu, etc., “Inserting 3D Projected Virtual Content into Broadcast Tennis Video”, ACM Multimedia 2006. C. Xu, etc., “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, 2004. 2015-9-18http://www.jdl.ac.cn7

8. Related work – Insertion Content Improve the ad effect Decrease intrusion VideoSense Textual relevance Local visual and aural relevance T. Mei, X-S. Hua, L. Yang, S. Li, “VideoSense-Towards Effective Online Video Advertising”, 16th ACM International Conference on Multimedia, pp: 1075-1084, 2007 2015-9-18http://www.jdl.ac.cn8

9. Outline Motivation Related work The proposed VCI system Experimental results Conclusions 2015-9-18http://www.jdl.ac.cn9

10. 2015-9-18http://www.jdl.ac.cn10

11. Temporal Attention Basic idea The more different a frame/shot/video clip is to the preceding ones, the more probable for it to be attended Measure Novelty 2015-9-18http://www.jdl.ac.cn11

12. 2015-9-18http://www.jdl.ac.cn12

13. HAS Detection Shot novelty The longer a shot is, the more it is probable to be attended 2015-9-18http://www.jdl.ac.cn13

14. 2015-9-18http://www.jdl.ac.cn14

15. 2015-9-18http://www.jdl.ac.cn15 Spatial Attention Analysis Static attention Spatio-temporal attention Motion saliency Static novelty

16. Static Saliency (1) Psychological basis Contrast Information theory Our method Contrast and information theory Calculation Property of receptive field 2015-9-18http://www.jdl.ac.cn16

17. Static Saliency (2) Perceptive unit Pixel/block Region Object Color quantization 2015-9-18http://www.jdl.ac.cn17

18. 2015-9-18http://www.jdl.ac.cn18 Static Saliency (3) Contrast Information density Saliency

19. 2015-9-18 http://www.jdl.ac.cn19 Motion Saliency Motion Vector Space  HSV color space

20. Static Novelty (1) 2015-9-18http://www.jdl.ac.cn20

21. 2015-9-18http://www.jdl.ac.cn21 Static Novelty (2) Static novelty: An event’s importance along temporal axis Distance: KL

22. 2015-9-18http://www.jdl.ac.cn22

23. Static LAR Detection 2015-9-18http://www.jdl.ac.cn23

24. Dynamic LAR Detection 2015-9-18http://www.jdl.ac.cn24

25. 2015-9-18http://www.jdl.ac.cn25

26. Affine transformation 2015-9-18http://www.jdl.ac.cn26

27. 2015-9-18http://www.jdl.ac.cn27 Global Motion Estimation

28. Outline Motivation Related work The proposed VCI system Experimental results Conclusions 2015-9-18http://www.jdl.ac.cn28

29. Experiment Data Set – Test Video 2015-9-18http://www.jdl.ac.cn29 No.VideoGenreShotTime 1Friends situation comedy 20011:25 2 Children at House situation comedy 20014:48 3 A Date with LuYu Interview20020:49 4 Adventure to the west Outdoor teleplay 20025:48 Sum---- 80072:50

30. 2015-9-18http://www.jdl.ac.cn30 Experiment Data Set -- Virtual Content

31. 2015-9-18http://www.jdl.ac.cn31 Temporal attention & HAS (1)

32. 2015-9-18http://www.jdl.ac.cn32 Temporal attention & HAS (2) Noticing rate: Consistency: the similarity between attention curve and noticing curve No.Consistency 10.75 20.79 30.84 40.82

33. 2015-9-18http://www.jdl.ac.cn33 Temporal attention & HAS (3) Relationship between noticing rate and attention value

34. 2015-9-18http://www.jdl.ac.cn34 Spatial attention & LAR Invited the users to evaluate the brands he/she has noticed in the video. rate of GOOD VideoGOODNEUTRALBAD 172.2519.138.62 270.8723.136.00 366.2525.008.75 470.3825.624.00 Mean69.9423.226.84 Variance 6.678.555.20

35. Static Insertion Demo

36. 2015-9-18http://www.jdl.ac.cn36 Dynamic Insertion Evaluation Subjective evaluation Criteria 1. Is the result’s deformation consistent with the scene? 2. Does the inserted VC follow the camera motion? 3. To what degree the user is satisfied with the result? Scores: 1  5

37. 2015-9-18http://www.jdl.ac.cn37 Dynamic Insertion Result

38. 2015-9-18http://www.jdl.ac.cn38 Conclusion Main contribution A generic virtual content insertion system. A new method of temporal attention and HAS detection A new method of spatial attention and LAR detection A dynamic insertion method Future work The attention change caused by content insertion The interaction between insertion time and place