https://github.com/zaeemzadeh/IPM Iterative Projection and Matching: Finding Structure-preserving Representatives and Its Application to Computer Vision Alireza Zaeemzadeh* , Mohsen Joneidi* , Nazanin Rahnavard, and Mubarak Shah Joint work between Communications and Wireless Networks Lab (CWNLAB) and Center for Research in Computer Vision (CRCV) University of Central Florida https://github.com/zaeemzadeh/IPM ∗ indicates shared first authorship.

Problem Selecting K distinct representative samples from M available data points (K << M), while preserving the structure.

Applications Active learning, Dataset summarization, Video summarization, ...

Challenges Computational complexity Robustness to outliers Generalization Methods based on convex-relaxation are not usually feasible on large datasets. Selection based on diversity is not robust to outliers The selection algorithm should work for different problems and datasets, without much effort

Intuition of Our Method Eigenfaces are not within data points. SVD is computationally expensive. Our algorithm selects according to spectrum.

Iterative Projection and Mapping

Iterative Projection and Mapping What does this mean mathematically? rank-K  factorization T is a subset of data samples \pi_T(A) is projection of all data on span of the selected samples. a rank-K  factorization V contains k rows of A; while there is no constraint on U  Assume we are to select one sample at a time, which is the best representation of all data. Since Problem (3) involves a combinatorial search and is not easy to tackle, let us modify (3) into two consecutive problems. The first sub-problem relaxes the constraint vk 2 A  in (3) to a moderate constraint kvk = 1 , First right singular value Select one sample at a time and the second sub-problem reimposes the underlying constraint. First right singular value first right singular vector

Theoretical Guarantees Linear complexity At least one sample with high correlation with the first singular vector exists. The first right singular vector is the least susceptible to noise compared to all the other singular vectors. The first singular vector can be calculated in linear time.

Theoretical Guarantees Robustness to noise At least one sample with high correlation with the first singular vector exists. The first right singular vector is the least susceptible to noise compared to all the other singular vectors. The first singular vector can be calculated in linear time.

Theoretical Guarantees Existence of a highly correlated sample with the first singular vector. At least one sample with high correlation with the first singular vector exists. The first right singular vector is the least susceptible to noise compared to all the other singular vectors. The first singular vector can be calculated in linear time.

Experiments Active Learning on UCF 101 Learning Using Representatives CMU Multi-PIE ImageNet Video summarization on UTE Egocentric

Task I: Active Learning Addresses the costly data labeling problem. Train the model Model Labeled Training Set Unlabeled Data Oracle Selection Extract features and/or uncertainty scores.

Task I: Active Learning Dataset: UCF-101 13,320 action instances from 101 human action classes The average duration of each video is about 7 seconds. Model: 3D ResNet18 architecture, pretrained on Kinetics-400 dataset Feature space: convolutional features from the last convolutional layer

Task I: Active Learning (UCF101) During the first few cycles, since the classifier is not able to generate reliable uncertainty score, uncertainty-based selection does not lead to a performance gain. On the other hand, IPM is able to select the critical samples and outperforms other methods.

Task I: Active Learning (UCF101) DS3 IPM Clean and Jerk DS3 IPM Kayaking In general, in the clips selected by IPM, the critical features of the action, such as barbell and kayak, are more visible and/or the bounding box for the action is bigger. Lifting and Kayaking Frames of the first selected representative

Task I: Active Learning (UCF101) Knitting, PlayingFlute 2D visualization of two classes of UCF-101 dataset.

Task II: Learning Using Representatives Find the representatives and use them for learning. If the samples contain enough information, performance will not deteriorate, saves computation and storage.

Task II: Learning Using Representatives Dataset: Multi-PIE 249 subjects, 9 poses, 20 illuminations, and two expressions 9×20×2 images per subject Feature space: 200 dimensional space using PCA.

Task II: Learning Using Representatives (GAN) Multi-view face generation using CR-GAN. Trained on reduced training set 9 images per subject And on the full dataset 360 images per subject.

Task II: Learning Using Representatives (GAN) Quantitative performance investigation: Identity similarities between the real and generated images 256 D Features from a ResNet18, trained on MS-Celeb-1M. Distances of features correspond to the face dissimilarity

Task II: Learning Using Representatives K-medoids DS3 IPM IPM selects from 10 different angles, while the selected images by DS3 and K-medoids contain repetitious angles.

Task II: Learning Using Representatives (ImageNet) Dataset: ImageNet 1000 classes 700-1300 images per class Feature space: 128 dimensional space using non-parametric instance discrimination (unsupervised) Only IPM and k-medoids were able to generate results.

Task II: Learning Using Representatives (ImageNet) Selecting 5 representatives per class IPM K-medoids IPM K-medoids

Task II: Learning Using Representatives (ImageNet) How well the selected images cover the space? K-NN classification accuracy using selected representatives Accuracy using all the labeled data ( 1.2M samples) is 46.86%

Task III: Video Summarization The goal is to select key clips and create a video summary, such that it contains the most essential contents of the video. Two minutes summarization of the first video of UTE Egocentric dataset (232 minutes long). 24 5-second clips are selected. The selected scenes cover the story of the whole video which is about 4 hours.

Task III: Video Summarization Dataset: UT Egocentric (UTE) dataset (contains 4 first-person videos of 3-5 hours of daily activities) Feature space: 1024-dimensional feature vectors extracted using GoogleNet. Feature Extraction Clustering Selection Video summary Video clip

Task III: Video Summarization F-measure and recall scores using ROUGE-SU metric.

Conclusions IPM: Iterative Projection and Matching Linear complexity w.r.t. number of data points. Robustness to outliers. No parameters for fine tuning. The superiority of IPM is shown in different applications.

Thank You