Yu-Gang Jiang, Yanran Wang, Rui Feng Xiangyang Xue, Yingbin Zheng, Hanfang Yang Understanding and Predicting Interestingness of Videos Fudan University, Shanghai, China AAAI 2013, Bellevue, USA, July 2013

Large amount of videos on the Internet – Consumer Videos, advertisement… Some videos are interesting, while many are not Motivation More interestingLess interesting Two Advertisements of digital products

Applications Web Video Search Recommendation System...

Predicting Aesthetics and Interestingness of Images – Datta et al. ECCV, 2006; Dhar et al. CVPR, 2011; N. Murray et al. CVPR, 2012… We are the first to explore the interestingness of Videos Related Work More interesting Less interesting ………

Flickr – source: Flickr.com Consumer Video – videos: 1200 (20 hrs in total) YouTube – source: Youtube.com Advertisement Video – videos: 420 (4.2 hrs in total) Two New Datasets

Collected by 15 interestingness-enabled queries – Top 10% of 400 as interesting videos; Bottom 10% as uninteresting – 80 videos per category/query Flickr Dataset

Collected by 15 ads queries on YouTube 10 human assessors (5 females, 5 males) – Compare video pairs Annotation Interface YouTube Dataset General observation: videos with humorous stories, attractive background music, or better professional editing tend to be more interesting

Aim: compare two videos and tell which is more interesting Visual features Audio features High-level attribute features Ranking SVM results Multi-modal fusion vs. Our Computational Framework

Feature Visual features Color Histogram SIFTHOGSSIMGIST Audio features MFCC Spectrogram SIFT Audio-Six High-level attribute features ClassemesObjectBankStyle Flower, Tree, Cat, Face… Rule of Thirds Vanishing Point Soft Focus Motion Blur Shallow DOF …

Prediction – Ranking SVM trained on our dataset Chi square kernel for histogram-like features RBF kernel for the others – 2/3 for training and 1/3 for testing Evaluation – Prediction accuracy The percentage of correctly ranked test video pairs Prediction & Evaluation

Prediction Accuracies(%) Visual Feature Results Flickr YouTube

Audio Feature Results Flickr YouTube Prediction Accuracies(%)

Attribute Feature Results Flickr YouTube Different from predicting Image Interestingness

Visual Attribute Prediction Accuracies(%) Visual+Audio+Attribute Results Flickr YouTube % 5.4% Audio Visual+Audio+Attribute Visual+Audio

Conducted a pilot study on video interestingness Built two datasets to support this study – Publicly available at: Evaluated a large number of features – Visual + audio features are very effective – A few features useful in image interestingness do not work in video domain (e.g., Style Attributes…) Summary

Thank you ! Datasets are available at: