Query Based Video Summarization Jacob Laurel jslaurel@uab.edu University of Alabama at Birmingham Aidean Sharghi aidean@knights.ucf.edu University of Central Florida I. Problem & Motivation III. Data Collection and Results IV. Algorithm Overview Automatically generate a comprehensive video summary for a set of egocentric videos using a Determinantal Point Process (DPP) algorithm. The algorithm should generate the summary based on a user-specified query (i.e. summarizing the video for concepts such as “Car”) Applications: Surveillance Law-enforcement Consumer electronics (‘GoPro®’, ‘Google Glass’) Dataset No query-specific video summary dataset exists, thus we augment an existing set for various user-input queries and provide ground truths for all cases Data Collection Procedure: Each video was sampled, and every set of 5 frames was used to create a “shot” given to MTurk workers All shots were annotated through a custom Amazon Mechanical Turk User-Interface, and each was given to 3 separate MTurk workers Shots were sorted by respective concept annotations A ground-truth summary for the entire video was generated for each of the 4 videos For each query pair the union of the ground truth video summary with the shots from each concept pair was taken This was then provided to AMT workers for summarization Tested our data set with a k-DPP algorithm, using a k of 2. Video is broken up into sets of 10 frames and then the DPP kernel is computed using SentiBank features for each set From each set 2 representative and distinct frames are chosen based on frame dissimilarity as well as SentiBank[2] features relating to the query V. Algorithm Results We provide extensive baseline results for our dataset For each of the 4 cases, 15 different Query Pairs were selected (60 per video) Example of a generated summary for the query pair “Men” + “Computer” II. Data Set Overview Data Results Average Inter-user agreement among all videos found to be good: 𝑈𝑠𝑒𝑟1⋂𝑈𝑠𝑒𝑟2⋂𝑈𝑠𝑒𝑟3 𝑈𝑠𝑒𝑟1⋃𝑈𝑆𝑒𝑟2⋃𝑈𝑠𝑒𝑟3 ≈0.33 Annotation Counts: Data Overview: 4 Egocentric videos (each of length ~3hr.) originally obtained from UTE data set[1] Each video contains shot-level annotations from a list of 48 concepts We define a shot to be a set of 5 frames, sampled at 1 fps Each video is summarized for pairs of queries for the following 4 cases Case 1) A Pair of queries such that the shot-level intersection is above a threshold, T (both concepts are present in several shots) Case 2) A Pair of queries such that the shot-level intersection is ∅ (both concepts are never found in the same shot), but both concepts are found throughout the video greater than T times Case 3) A Pair of queries such that one occurs less than T times, but the other occurs greater than T times Case 4) A Pair of queries such that both are missing from the video (both occur less than T times) Automatically Generated Summary . . . User’s Query: Men + Computer VII. Future Work Expand data set to include non-egocentric videos, and expand upon other Video-Summarization algorithms to improve benchmarks Concept and Query Pair Selection: Concepts selected by taking most popular Youtube and Vine queries and comparing with SentiBank[2] concepts to select a comprehensive set (48 were ultimately chosen) Query Pairs selected as follows: Case 1) Pairs were sampled with weighting given to the intersection count 𝐶𝑜𝑢𝑛𝑡 𝑄 1 ⋂ 𝑄 2 = { 𝑆ℎ𝑜𝑡 1 … 𝑆ℎ𝑜𝑡 𝑁 } 𝕝 𝑄 1 𝕝(𝑄 2 ) 𝑃(𝑄𝑢𝑒𝑟𝑦1,𝑄𝑢𝑒𝑟𝑦2)∝ 𝐶𝑜𝑢𝑛𝑡 𝑄 1 ⋂ 𝑄 2 Case 2) Pairs were sampled with weighting given as follows 𝐶𝑜𝑢𝑛𝑡 𝑄 1 = { 𝑆ℎ𝑜𝑡 1 … 𝑆ℎ𝑜𝑡 𝑁 } 𝕝( 𝑄 1 ) 𝐶𝑜𝑢𝑛𝑡 𝑄 2 = { 𝑆ℎ𝑜𝑡 1 … 𝑆ℎ𝑜𝑡 𝑁 } 𝕝( 𝑄 2 ) 𝑃(𝑄𝑢𝑒𝑟𝑦1,𝑄𝑢𝑒𝑟𝑦2)∝ 𝐶𝑜𝑢𝑛𝑡 𝑄 1 ⦁ 𝐶𝑜𝑢𝑛𝑡 𝑄 2 𝐶𝑜𝑢𝑛𝑡 𝑄 1 + 𝐶𝑜𝑢𝑛𝑡 𝑄 2 Cases 3 and 4) Pairs were sampled uniformly Example Ground-Truth Summary for “Car” query: VIII. References Input Video Query + Y. J. Lee, J. Ghosh, and K. Grauman, “Discovering important people and objects for egocentric video summarization,” 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012. Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel and Shih-Fu Chang. "Large-scale Visual Sentiment Ontology and Detectors Using Adjective Noun Pairs," ACM Multimedia Conference, Barcelona, Oct 2013. … … … IX. Acknowledgements We wish to thank Dr. Shah, Dr. Gong, and Dr. Lobo for their support and mentorship throughout the Summer 2016 REU Program. User-selected Summary Shots