Watch Listen & Learn: Co-training on Captioned Images and Videos Sonal Gupta, Joohyun Kim, Kirstenn Garuman and Raymond Mooney.

Their endeavour : Recognize categories from natural scenes from images with captions Recognizing human actions in sports videos accompanied with commentary.

How do they go about it Their model learns to classify images and videos from labelled and unlabelled multi- modal examples. A semi-supervised approach. Use image or video content together with its textual annotation(captions or commentary) to learn scene and action categories.

Features: Visual Features Textual features Static Image features Motion descriptors from videos Textual features

Histogram Of Oriented Gradients(HOG) Divide image into small connected regions(cells) Compile a histogram of gradient directions or edge orientation(This is the descriptor). The implementation of these descriptors can be achieved by dividing the image into small connected regions, called cells, and for each cell compiling a histogram of gradient directions or edge orientations for the pixels within the cell. The combination of these histograms then represents the descriptor.

Gabor Filter Convolution of a sinusoid with a gaussian Texture detection

LAB color space L(lightness) a and b are color opponent dimensions Includes all perceivable colors(exceeds that of RGB and CYMk) device independent Designed to approximate human vision

Static Image Features Each Image is broken into 4x6 grids texture feature for each regions using gabor filters Record the mean, std deviation & skewness of per channel RGB Lab color pixel values 30 -dimensional feature vector for each of the 24 regions K-means clustering Each region of each image is assigned to the closest cluster centroid.

N x 30

Bag of visual words The final bag of visual words for each image consists of: A vector of k-values: The ith element represents the number of regions in the image that belongs to the ith cluster.

Motion descriptors laptev spatio temporal motion descriptors At each feature point the patch is divided into 3x3x2 spatio-temporal blocks 4-bin HOG descriptors calculated for each block 72 element feature vector is obtained. http://www.irisa.fr/vista/Papers/2005_ijcv_laptev.pdf A video clip is represented as a histogram over this vocabulary Motion descriptors from all video in the training set are clustered to form a vocabulary.

N x 72

Textual features Image captions or transcribed video commentary Preprocess:Remove stop words and stem the remaining. The frequency of resulting word stem comprised the feature set.

Co- training What is it? - A semi supervised learning paradigm that exploits two mutually independent views. Independent views in our case: Text classifier Visual classifier

Text classifier Visual Classifier Text View Visual View + - Initially labelled Instances

Supervised learning Classifiers used to learn from labeled instances Visual Classifier Text classifier Text View + _ Visual View - + Initially labelled Instances

Co -train The trained classifier from the previous step used to label unlabelled instances Text classifier Visual Classifier Text View Visual View Unlabelled Instances

Supervised learning Classify Most confident instances Visual View Visual Classifier Text classifier Text View Visual View Partially labelled Instances

Supervised learning Label all views in the instances Visual Classifier Text classifier Text View + - Visual View + - Classifier labelled Instances

Retrain Classifier Retrain based on the new labels Visual Classifier Text classifier Text View + - Visual View + -

Classify a new instance Text View Visual View Text View Visual view Text View Visual View Text view Visual View

Ready for Co- training Classify Most confident instances System Input A Set of labelled and unlabeled examples each with two set of features(one for each view) Two classifiers whose predictions can be combined to classify new test instances. System Input Output

Combined result used for labeling test instances Early and Late Fusion Early Fusion Visual Features Fused Vector Classifier Combined result used for labeling test instances Textual Features

Combined result used for labeling test instances Early and Late Fusion Late Fusion Visual Features Visual Classifier Combined result used for labeling test instances Textual Features Text Classifier

Semi-supervised EM and Transductive SVMs Semi-supervised Expectation Maximization Learns the probabilistic classifier from the labeled training data Performs EM iterations E-step – uses the currently trained classifier to probabilistically label the unlabeled training examples M-step – retrains the classifier on the union of labeled data and probabilistically labeled unsupervised examples

Transductive SVMs Method of improving the generalization accuracy of SVMs by using unlabeled data Finds the labeling of the test examples that results in maximum margin hyperplane that separates the positive and negative examples of both training and testing data

Transductive SVMs (contd) Transductive SVMs are typically designed to improve the performance of the test data by utilizing its availability during training. It is also used directly in semi supervised setting where unlabeled data is available during training, which comes from the same distribution as the test data.

Methodology For co-training, Support Vector Machine – Base classifier for both image and text views We use Weka implementation of sequential minimal optimization (SMO) SMO is an algorithm for efficiently solving the optimization problem which arises during the training of SVMs

Methodology (Continued) Parameters used in SMO -RBF Kernel (γ=0.01) -Batch size: 5 Confidence Threshold Static Images Videos Image/Video view 0.65 0.6 Text view 0.98 0.9

Methodology (Continued) Ten iterations of ten-fold cross validation is performed to get smoother and reliable results. Test set is disjoint from both the labeled and unlabeled training data. Learning curves are used for evaluating the accuracy.

Learning Curves The learning curve can represent at a glance, the initial difficulty of learning something and, to an extent, how much there is to learn after initial familiarity. These curves are generated where at each point some fraction of the training data is labeled and the remainder is used as unlabeled training data

Results Classify captioned static images: Image dataset : Israel dataset Images have short text captions Two classes : ‘Desert’ , ‘Trees’

Examples of images DESERT Ibex in Judean Desert Bedouin Leads His Donkey That Carries Load Of Straw TREES Ibex Eating In The Nature Entrance To Mikveh Israel Agricultural School

Co-training Vs Supervised Classifiers

Co-training Vs. Semi-Supervised EM Co-training Vs. Transductive SVM

Results(contd.) Recognize actions from Commented videos: Video clips of soccer and ice-skating Resized to 240x360 resolution and then divided manually into short clips. Clip length varies from 20 to 120 frames. Four categories : kicking, dribbling, spinning, dancing

Examples of Videos

Examples of Videos(contd.)

Co-training Vs. supervised learning on commented video dataset

Co-training Vs. supervised learning when text commentary is not available

Limitations in the approach Data set used is small and requires only binary classification. Images that have explicit captions are used.

QUESTIONS ?????

THANK YOU Joydeep Sinha Anuhya Koripella Akshitha Muthireddy