1. © Arjen P. de Vries 2001 1 Arjen P. de Vries arjen@acm.org Fascinating Relationships between Media and Text

2. © Arjen P. de Vries 2001 2 From Video to a List of Images… …

3. © Arjen P. de Vries 2001 3 Query image

4. © Arjen P. de Vries 2001 4 Feature Space Indexing N-dimensional space Feature extraction Ranking Display

5. © Arjen P. de Vries 2001 5 Complicating Factors What are Good Feature Models? What are Good Ranking Functions? Subjective Queries

6. © Arjen P. de Vries 2001 6 Mirror Approach (1/2) Gather User’s Knowledge –Introduce semi-automatic processes for selection and combination of feature models Local Information –Relevance feedback from a user Global Information –Thesauri constructed from all users

7. © Arjen P. de Vries 2001 7 Mirror Approach (2/2) N-dimensional space Feature extraction ClusteringRanking Concepts Display Thesauri

8. © Arjen P. de Vries 2001 8 Visual Thesaurus Correct cluster representing ‘Tree’, ‘Forest’ Glcm_47 ‘Incoherent’ cluster Fractal_23 Mis-labeled cluster Gabor_20

9. © Arjen P. de Vries 2001 9 Mirror Results Thesaurus Only After Feedback

10. © Arjen P. de Vries 2001 10 ‘Explaining’ the Results Paivio’s dual coding theory conjectures that the human brain processes textual terms (logogens) as well as image terms (imagens) Term frequency and inverse document frequency intuitively express the amount of surprise of observing the event represented by the term Also matches similar music: grunge, house, … Even works for predicting avalanches!

11. © Arjen P. de Vries 2001 11 Query Articulation Problem N-dimensional space Feature extraction How to articulate the query?

12. © Arjen P. de Vries 2001 12 What is the query semantics ?

13. © Arjen P. de Vries 2001 13 More than Sub-Image Search Irrelevant Background Relevant Colors Distinguishing Shapes …

14. © Arjen P. de Vries 2001 14 Irrelevant Background QueryResult

15. © Arjen P. de Vries 2001 15 Known Item

16. © Arjen P. de Vries 2001 16 Query

17. © Arjen P. de Vries 2001 17 Results …

18. © Arjen P. de Vries 2001 18 Query

19. © Arjen P. de Vries 2001 19 Results …

20. © Arjen P. de Vries 2001 20 Problem Video TREC query images are quite different from the (low-quality) video data in the collection The current translation model from collection objects to query objects is incorrect Figure out better ways, possibly using: –Color thesaurus, transitions between colors –Explicit background and foreground models –…

21. © Arjen P. de Vries 2001 21 Summarizing Don’t give up… But… Stay Realistic!