A Generic Virtual Content Insertion System Based on Visual Attention Analysis H. Liu 1, 2, S. Jiang 1, Q. Huang 1, 2, C. Xu 2, 3 1 Institute of Computing Technology, Chinese Academy of Sciences 2 China-Singapore Institute of Digital Media 3 National Lab of Pattern Recognition, Institute of Automation

Outline Motivation Related work The proposed Virtual Content Insertion (VCI) system Experimental results Conclusions http://

Virtual Content Insertion http:// Convenient Changeable Cost less To construct a generic VCI system

Challenge Advertisement insertion VS. Augmentation Software based VS. Hardware based Challenge Insertion time Insertion place Insertion method Insertion content

Related work– Insertion Time Insert advertisements into video prologue Be neglected Insert the ad into interesting segments Our method Temporal attention Higher Attentive Shots K. Wan, C. Xu, “Automatic Content Placement in Sports Highlights”, ICME, http://

Related work– Insertion Place Static region Color consistent region Visual relevance measure Lower informative region Our method Spatial attention Lower Attentive Region C. Xu, etc., “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, K. Wan, etc., “Automatic Content Placement in Sports Highlights”, ICME, Y. Li, etc., “Real Time Advertisement Insertion in Baseball Video Based on Advertisement Effect”, ACM Multimedia, http://

Related work – Insertion Method Challenge Camera parameters unknown Existing methods Structure of the scene Predefined landmarks Our method Affine transformation Global Motion Estimation X. Yu, etc., “Inserting 3D Projected Virtual Content into Broadcast Tennis Video”, ACM Multimedia C. Xu, etc., “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, http://

Related work – Insertion Content Improve the ad effect Decrease intrusion VideoSense Textual relevance Local visual and aural relevance T. Mei, X-S. Hua, L. Yang, S. Li, “VideoSense-Towards Effective Online Video Advertising”, 16th ACM International Conference on Multimedia, pp: , http://

Outline Motivation Related work The proposed VCI system Experimental results Conclusions http://

http://

Temporal Attention Basic idea The more different a frame/shot/video clip is to the preceding ones, the more probable for it to be attended Measure Novelty http://

http://

HAS Detection Shot novelty The longer a shot is, the more it is probable to be attended http://

http://

http:// Spatial Attention Analysis Static attention Spatio-temporal attention Motion saliency Static novelty

Static Saliency (1) Psychological basis Contrast Information theory Our method Contrast and information theory Calculation Property of receptive field http://

Static Saliency (2) Perceptive unit Pixel/block Region Object Color quantization http://

http:// Static Saliency (3) Contrast Information density Saliency

Motion Saliency Motion Vector Space  HSV color space

Static Novelty (1) http://

http:// Static Novelty (2) Static novelty: An event’s importance along temporal axis Distance: KL

http://

Static LAR Detection http://

Dynamic LAR Detection http://

http://

Affine transformation http://

http:// Global Motion Estimation

Outline Motivation Related work The proposed VCI system Experimental results Conclusions http://

Experiment Data Set – Test Video http:// No.VideoGenreShotTime 1Friends situation comedy 20011:25 2 Children at House situation comedy 20014:48 3 A Date with LuYu Interview20020:49 4 Adventure to the west Outdoor teleplay 20025:48 Sum :50

http:// Experiment Data Set -- Virtual Content

http:// Temporal attention & HAS (1)

http:// Temporal attention & HAS (2) Noticing rate: Consistency: the similarity between attention curve and noticing curve No.Consistency

http:// Temporal attention & HAS (3) Relationship between noticing rate and attention value

http:// Spatial attention & LAR Invited the users to evaluate the brands he/she has noticed in the video. rate of GOOD VideoGOODNEUTRALBAD Mean Variance

Static Insertion Demo

http:// Dynamic Insertion Evaluation Subjective evaluation Criteria 1. Is the result’s deformation consistent with the scene? 2. Does the inserted VC follow the camera motion? 3. To what degree the user is satisfied with the result? Scores: 1  5

http:// Dynamic Insertion Result

http:// Conclusion Main contribution A generic virtual content insertion system. A new method of temporal attention and HAS detection A new method of spatial attention and LAR detection A dynamic insertion method Future work The attention change caused by content insertion The interaction between insertion time and place