Video Summarization via Determinantal Point Processes (DPP) Boqing Gong University of Southern California Joint work with Wei-Lun Chao, Kristen Grauman, and Fei Sha

Background Basic idea of DPP Sequential DPP (NIPS 2014) Large-margin training of DPP Conclusion

Background Motivation: Representation: Subset Selection problem Indispensable for fast video browsing and retrieval Representation: Key frames / segments extraction Subset Selection problem

Background Video summarization is hard: Naïve solution: Clustering Individual selected frame: Representativeness Selected frames as a whole: Diversity Naïve solution: Clustering Competing !

Background Clustering works?

Video summarization: an overview Video summarization is hard: What criteria lead to user perspective? What kind of models: Supervised learning ! Diverse subset with representative items

Background How to model subset selection problem? Structured prediction, submodular functions Determinantal Point Processes (DPPs) [Alex Kulesza and Ben Taskar, 2012]

Background Basic idea of DPP Sequential DPP (NIPS 2014) Large-margin training of DPP Conclusion

Basic idea of DPP Idea: A point process based on matrix determinant. Formulation: M discrete items (binary decision)

Basic idea of DPP Why diverse? Extreme cases:

Basic idea of DPP Learning in DPP: 11

Background Basic idea of DPP Sequential DPP (NIPS 2014) Large-margin training of DPP Conclusion

Sequential DPP Motivation: Proposed Idea: The temporal structure of video is missing Proposed Idea: Sequential DPP via Markov properties

… …

Sequential DPP Modeling the sequential structure: Conditional DPP: still a DPP !

Sequential DPP Parameterization:

Inference and Learning Allow brute-force search in small chunks Optimization:

Sequential DPP Experimental setting: 3 datasets: OVP (50), Youtube (39), Kodak (18) Fisher vectors + Saliency + Contextual features Evaluation: Recall, Precision, and F1 score Comparison: unsupervised methods & vanilla DPP

Sequential DPP Experimental Results:

Sequential DPP Experimental Results:

Sequential DPP Experimental Results:

Background Basic idea of DPP Sequential DPP (NIPS 2014) Large-margin training of DPP Conclusion

Learning parameters in DPP Maximum likehood estimation Focuses on observed data only Large-margin training Maximizes margin between observed and undesired data Discriminative learning More flexible: incorporating evaluation metrics

Large-margin training of DPP More discriminative and flexible

Conclusion Supervised learning for video summarization DPPs: modeling diversity subset selection Video structure: Sequential DPP Parameterization: Neural networks Future work Better inference algorithms Models beyond DPP (submodular)