ICASSP, May Arjen P. de Vries Thijs Westerveld Tzvetanka I. Ianeva Combining Multiple Representations on the TRECVID Search Task

ICASSP, May Introduction Video Retrieval should take advantage of information from all available sources and modalities –…but so far ASR best for almost any query Combining information sources –Different models/modalities –Multiple example images

ICASSP, May ‘Language Modelling’ approach to IR DocsModels

ICASSP, May Calculate conditional probabilities of observing query samples given each model in the collection Retrieval Models P(Q|M 1 ) P(Q|M 4 ) P(Q|M 3 ) P(Q|M 2 ) Query

ICASSP, May Static Model Indexing –Estimate a Gaussian Mixture Model from each keyframe (using EM) –Fixed number of components (C=8) –Feature vectors contain colour, texture, and position information from pixel blocks:

ICASSP, May Dynamic Model Indexing: GMM of multiple frames (N=29) around keyframe Feature vectors extended with time- stamp in [0,1]: 0.5 1

ICASSP, May Dynamic Model

ICASSP, May Dynamic Model Advantages More training data for models Reduced dependency upon selecting appropriate keyframe Some spatio-temporal aspects of shot are captured –(Dis-)appearance of objects

ICASSP, May Experimental Set-up Build models for each shot –Static, Dynamic, Language Build Queries from topics –Construct simple keyword text query –Select visual example –Rescale and compress example images to match video size and quality

ICASSP, May Combining Modalities Independence assumption textual/visual –P(Q t,Q v |Shot) = P(Q t |LM) * P(Q v |GMM) Combination works if both runs useful [CWI:TREC:2002] Dynamic run more useful than static run RunMAP ASR only.130 Static only.022 Static+ASR.105 Dynamic only.022 Dynamic+ASR.132

ICASSP, May Combining Modalities Dynamic: Higher Initial Precision

ICASSP, May Dow Jones Topic (120)

ICASSP, May Dow Jones Topic (120) “Dow Jones Industrial Average rise day points” + =

ICASSP, May Dow Jones Topic (120)

ICASSP, May Arafat topic (103)

ICASSP, May Arafat Topic (103)

ICASSP, May Basketball topic (101) Baseball topic (102)

ICASSP, May Basketball Topic

ICASSP, May Merging Run Results

ICASSP, May Merging Run Results Combining (conflicting) examples difficult [CWI:TREC:2002] Single example  Miss relevant shots Round-Robin Merging Combined

ICASSP, May Merging Run Results Combining (conflicting) examples difficult [CWI:TREC:2002] Single example  Miss relevant shots Round-Robin Merging Combined ASR Single All Selected Best

ICASSP, May Flames (112)

ICASSP, May Flames Topic (112)

ICASSP, May Conclusions For most topics, neither the static nor the dynamic visual model captures the user information need sufficiently… …averaged over 25 topics however, it is better to use both modalities than ASR only Working hypothesis: Matching against both modalities gives robustness

ICASSP, May Conclusions Dynamic captures visual similarity better –Thanks to spatio-temporal aspects? Experiments with full covariance matrix for -dims Static model of KF is too fragile –Dependency on single KF? To be tested by ranking max(all I-frames in shot) –Not enough training data?

ICASSP, May Conclusions Visual aspects of an information need are best captured by using multiple examples Combining results for multiple (good) examples in round-robin fashion, each ranked on both modalities, gives near- best performance for almost all topics