1. Finding Better Answers in Video Using Pseudo Relevance Feedback Informedia Project Carnegie Mellon University Carnegie Mellon Question Answering from Errorful Multimedia Streams ARDA AQUAINT

2. Carnegie Mellon 2 Outline Pseudo-Relevance Feedback for Imagery Experimental Evaluation Results Conclusions

3. Carnegie Mellon 3 Motivation Question Answering from multimedia streams Questions contain text and visual components Want a good image that represents the ‘answer’ Improve performance of images retrieved as answers Relevance feedback works for text retrieval !

4. Carnegie Mellon 4 Finding Similar Images by Color

5. Carnegie Mellon 5 Finding Similar Scenes

6. Carnegie Mellon 6 Similarity Challenge: Images containing similar content

7. Carnegie Mellon 7 What is Pseudo Relevance Feedback Relevance Feedback (Human intervention) QUERY SYSTEMRESULTS Relevance Judgment HUMAN Feedback Why Pseudo? QUERY SYSTEMRESULTS Feedback without human intervention

8. Carnegie Mellon 8 Original System Architecture Simply weighted linear combination of video, audio and text retrieval score Retrieval Agents Query TextImage Text ScoreImage Score Final Score

9. Carnegie Mellon 9 System Architecture with PRF New step: Classification through Pseudo Relevance Feedback (PRF) Combine with all other information agents (text, image) Query TextImage Final Score Retrieval Agents Text Score Image Score PRF Score

10. Carnegie Mellon 10 Classification from Modified PRF Automatic retrieval technique Modification: use negative data as feedback Step-by-step Run base retrieval algorithm on image collection K-Nearest neighbor (KNN) on color and texture Build classifier Negative examples: least relevant images in the collection Positive examples: image queries Classify all data in the collection to obtain ranked results

11. Carnegie Mellon 11 The Basic PRF Algorithm for Image Retrieval Input Query Examples q 1 … q n Target Examples t 1 … t n ========================= Output Final score F i and final ranking for every target t i ========================= Algorithm Given initial score s 0 i for each ti based on f 0 (t i, q 1 … q n ) Using an initial similarity measure f 0 as a base Iterate k = 1 … max Given score s k i, sample positive instances p k i and negative instances n k i using sampling strategy S Compute updated retrieval score s i k+1 = f i k+1 (t i ) where f i k+1 is trained/learned using n k i,p k i Combine all scores for final score F i =g(s 0 … s max )

12. Carnegie Mellon 12 Analysis: PRF on Synthetic Data

13. Carnegie Mellon 13 PRF on Synthetic Data

14. Carnegie Mellon 14 Evaluation using the 2002 TREC Video Retrieval Task Independent collection, queries, relevant results available Search Collection Total Length: 40.16 hours MPEG-1 format Collected from Internet Archive and Open Video websites, documentaries from the ‘50s 14,000 shots 292,000 I-frames (images) Query 25 queries Text, Image(Optional), Video(Optional)

15. Summary of ’02 Video Queries

16. Carnegie Mellon 16 Analysis of Queries (2002) Specific item or person Eddie Rickenbacker, James Chandler, George Washington, Golden Gate Bridge, Price Tower in Bartlesville, OK Specific fact Arch in Washington Square Park in NYC, map of continental US Instances of a category football players, overhead views of cities, one or more women standing in long dresses Instances of events/activities people spending leisure time at the beach, one or more musicians with audible music, crowd walking in an urban environment, locomotive approaching the viewer

17. Carnegie Mellon 17 Sample Query and Target Query: Find pictures of Harry Hertz, Director of the National Quality Program, NIST

18. Carnegie Mellon 18 Sample Query and Target Query: Find pictures of Harry Hertz, Director of the National Quality Program, NIST Speech: We’re looking for people that have a broad range of expertise that have business knowledge that have knowledge on quality management on quality improvement and in particular … OCR: H,arry Hertz a Director aro 7 wa-,i,,ty Program,Harry Hertz a Director

19. Carnegie Mellon 19 Example Images

20. Carnegie Mellon 20 Example Images Selected for PRF

21. Carnegie Mellon 21 Combination of Agents Multiple Agents Text Retrieval Agent Base Image Retrieval Agent Nearest Neighbor on Color Nearest Neighbor on Texture Classification PRF Agent Combination of multiple agents Convert scores to posterior probability Linear combination of probabilities

22. Carnegie Mellon 22 2002 Results *Video OCR was not relevant in this collection MethodPrecisionRecallMAP Speech Transcripts only0.03480.14450.0724 SR + Color/Texture0.08920.2200.1046 SR + Color/Texture + PRF0.09240.2160.1124

23. Carnegie Mellon 23 Distance Function for Query 75

24. Carnegie Mellon 24 Distance Function for Query 89

25. Carnegie Mellon 25 Effect of Pos/Neg Ratio and Combination Weight

26. Carnegie Mellon 26 Selection of Negative Images + Combination

27. Carnegie Mellon 27 Discussion & Future Work Discussion Results are sensitive to queries with small numbers of answers Images alone cannot fully represent the query semantics Future Work Incorporate more agents Utilize the relationship between multiple agent information Better combination scheme Include web image search (e.g. Google) as query expansion

28. Carnegie Mellon 28 Conclusions Pseudo-relevance feedback works for text retrieval This is not directly applicable to image retrieval from video due to low precision in the top answers Negative PRF was effective for finding better images