1. Query-Focused Video Summarization – Week 1
Jacob Laurel
Mentors: Aidean Sharghi and Dr. Boqing Gong
UCF CRCV REU 2016

2. Project Proposal
Goal: Summarize a video with a diverse subset of frames (meaning no redundancy). Also frames will be annotated so that a semantic summary will also be produced by the algorithm.
Annotate a video data set to use for testing and training
Possible applications of this project: Egocentric video summarization (i.e. Google Glass) , surveillance footage summarization, video search engine recommendations, pose estimation

3. Determinantal Point Process
Probabilistic process
Motivation: We wish to randomly take a diverse subset A, of some set S
We want to formulate our subset such that elements with diverse/different features are most likely (Intuitively, elements with similar features “repel” each other)
Such probabilities can be computed via determinants of a matrix (hence name)

4. Fig. 3. A diverse subset of frames from a video
Illustration of DPPs
Fig 1. MATLAB demo of a SDPP
Fig 2. Graphical illustration of points uniformly sampled (left) and distributed according to a DPP (right)
Fig. 3. A diverse subset of frames from a video

5. Step 1) Data Preparation
Annotate and prepare videos taken from UT Egocentric dataset
This will be done in conjunction with a GUI that can specify semantic information about the frames (used when querying)

6. Step 2) Design of the DPP
Our Design will follow a sequential structure
Different Neural Network configurations can be used to learn the best kernel matrix
Existing structure incorporates a single hidden layer
Different image features will be experimented with to determine the best model. Potential directions (as of 5/26)
SIFT features
GIST global image descriptor
Visual Bag of words model
We also propose to incorporate semantic features into the feature vector

7. Step 3) Experimental Procedure
The DPP will be tested using various feature vectors in OpenCV and Matlab
Neural Network to learn the kernel matrix will be constructed in Keras or Caffe
Model will need to be trained from scratch since no existing weights for this application are readily available
Scope will focus on feed-forward Neural Networks, as opposed to CNN’s to reduce computational requirements and allow for more degrees of freedom for other areas

8. Checklist for this week:
Downloaded all necessary software
Familiarized with Keras and reproduced simple NN’s
Ran and edited MATLAB DPP code
Familiarized with existing literature and downloaded the Data set from UT’s website
Reproduced simple version of Dr. Gong’s paper

9. Results from implementing a simple DPP
Fig. 5. Original video
Fig. 4. Subset of images generated by the DPP kernel

10. Next Week To-do Annotate Data set, with short descriptions
Generate Ground truth summaries
Compare method with other state-of-the-art approaches