ICME 2004 Tzvetanka I. Ianeva Arjen P. de Vries Thijs Westerveld A Dynamic Probabilistic Multimedia Retrieval Model

ICME 2004 Introduction Video Representation schemes used for retrieval: –Static –Spatio-temporal Video is a temporal media so a ‘good’ model solves the limitations of keyframe-based shot representation

ICME 2004 Spatio-temporal grouping Spatial priority and tracking of regions from frame to frame Joint spatial and temporal segmentation –Human vision finds salient structures jointly in space and time (Gepshtein and Kubovy, 2000)

ICME 2004 Motivation Pursue video retrieval instead of image (keyframe) retrieval Extension of the Static Probabilistic Multimedia Retrieval model (2003) GMM in DCT-space-time domain –Diagonal covariance

ICME 2004 Static Model DocsModels Indexing - Estimate Gaussian Mixture Models from images using EM - Based on feature vector with colour, texture and position information from pixel blocks - Fixed number of components

ICME 2004 Static Model Indexing –Estimate a Gaussian Mixture Model from each keyframe (using EM) –Fixed number of components (C=8) –Feature vectors contain colour, texture, and position information from pixel blocks:

ICME 2004 Static Model Models P(Q|M 1 ) P(Q|M 4 ) P(Q|M 3 ) P(Q|M 2 ) Query Retrieval –Calculate conditional probabilities of query samples given models in collection

ICME 2004 Dynamic Model Selecting frames – 1 second sequence around the keyframe – Entire video shot as sequence of frames sampled at regular intervals Features

ICME 2004 Dynamic Model Indexing: GMM of multiple frames around keyframe Feature vectors extended with time- stamp normalized in [0,1]: 0.5 1

ICME 2004 Dynamic Model

ICME 2004 Query example: A single image Artificial sequence of 29 images as the single query example where the time is normalized between 0 and 1 Extend the query example image’s features with a fixed temporal feature value of 0.5 – Better results and lower computational cost

ICME 2004 Dynamic Model Advantages More training data for models –Less sensitive to random initialization Reduced dependency upon selecting appropriate keyframe Some spatio-temporal aspects of shot are captured –(Dis-)appearance of objects

ICME 2004 Dynamic Model

ICME 2004 Dynamic Model

ICME 2004 Dynamic Model

ICME 2004 Retrieval Framework Smoothing Building dynamic GMMs Likelihood goes to infinity ???

ICME 2004 Experimental Set-up Build models for each shot –Static, Dynamic, Language Build Queries from topics –Construct simple keyword text query –Select visual example –Rescale and compress example images to match video size and quality

ICME 2004 Combining Modalities Independence assumption textual/visual –P(Q t,Q v |Shot) = P(Q t |LM) * P(Q v |GMM) Combination works if both runs useful [CWI:TREC:2002] Dynamic run more useful than static run RunMAP ASR only.130 Static only.022 Static+ASR.105 Dynamic only.022 Dynamic+ASR.132

ICME 2004 Combining Modalities Dynamic: Higher Initial Precision

ICME 2004 Dynamic: Higher initial precision Static run Dynamic run

ICME 2004 Dow Jones Topic (120)

ICME 2004 Dow Jones Topic (120) “Dow Jones Industrial Average rise day points” + =

ICME 2004 Conclusions Dynamic model captures visual similarity better –Spatio-temporal aspects –More training data –Apropriate key-frame less critical –Less sensitive to the random initialization ASR + dynamic better than either alone

ICME 2004 Future work More data needs more computation effort – optimizations ? Avoid the singular solutions Dynamic number of components ? Full covariance in space-time Integration of audio

ICME 2004 Thanks !!!

ICME 2004 Merging Run Results Combining (conflicting) examples difficult [CWI:TREC:2002] Single example  Miss relevant shots Round-Robin Merging Combined

ICME 2004 Merging Run Results

ICME 2004 Merging Run Results Combining (conflicting) examples difficult [CWI:TREC:2002] Single example  Miss relevant shots Round-Robin Merging Combined ASR Single All Selected Best

ICME 2004 Conclusions Visual aspects of an information need are best captured by using multiple examples Combining results for multiple (good) examples in round-robin fashion, each ranked on both modalities, gives near- best performance for almost all topics