© Arjen P. de Vries Arjen P. de Vries Fascinating Relationships between Media and Text
© Arjen P. de Vries From Video to a List of Images… …
© Arjen P. de Vries Query image
© Arjen P. de Vries Feature Space Indexing N-dimensional space Feature extraction Ranking Display
© Arjen P. de Vries Complicating Factors What are Good Feature Models? What are Good Ranking Functions? Subjective Queries
© Arjen P. de Vries Mirror Approach (1/2) Gather User’s Knowledge –Introduce semi-automatic processes for selection and combination of feature models Local Information –Relevance feedback from a user Global Information –Thesauri constructed from all users
© Arjen P. de Vries Mirror Approach (2/2) N-dimensional space Feature extraction ClusteringRanking Concepts Display Thesauri
© Arjen P. de Vries Visual Thesaurus Correct cluster representing ‘Tree’, ‘Forest’ Glcm_47 ‘Incoherent’ cluster Fractal_23 Mis-labeled cluster Gabor_20
© Arjen P. de Vries Mirror Results Thesaurus Only After Feedback
© Arjen P. de Vries ‘Explaining’ the Results Paivio’s dual coding theory conjectures that the human brain processes textual terms (logogens) as well as image terms (imagens) Term frequency and inverse document frequency intuitively express the amount of surprise of observing the event represented by the term Also matches similar music: grunge, house, … Even works for predicting avalanches!
© Arjen P. de Vries Query Articulation Problem N-dimensional space Feature extraction How to articulate the query?
© Arjen P. de Vries What is the query semantics ?
© Arjen P. de Vries More than Sub-Image Search Irrelevant Background Relevant Colors Distinguishing Shapes …
© Arjen P. de Vries Irrelevant Background QueryResult
© Arjen P. de Vries Known Item
© Arjen P. de Vries Query
© Arjen P. de Vries Results …
© Arjen P. de Vries Query
© Arjen P. de Vries Results …
© Arjen P. de Vries Problem Video TREC query images are quite different from the (low-quality) video data in the collection The current translation model from collection objects to query objects is incorrect Figure out better ways, possibly using: –Color thesaurus, transitions between colors –Explicit background and foreground models –…
© Arjen P. de Vries Summarizing Don’t give up… But… Stay Realistic!