1. Chapter 11 Beyond Bag of Words

2. Question Answering n Providing answers instead of ranked lists of documents n Older QA systems generated answers n Current QA systems extract answers from large corpora such as the Web n Fact-based QA limits range of questions to those with simple, short answers  Who, When, Where, What, Which, How (5W1H) questions 2

3. QA Architecture 3

4. Fact-Based QA n Questions are classified by type of answer expected  most categories correspond to named entities n Category is used to identify potential answer passages n Additional NLP to rank passages and identify answer 4

5. 5 Fact-Based QA

6. Other Media n Many other types of information are important for search applications  e.g., scanned documents, speech, music, images, video n Typically there is no associated text  although user tagging is important in some applications n Retrieval algorithms can be specified based on any content-related features that can be extracted, e.g., captions of pictures 6

7. Noisy Text n Optical character recognition (OCR) and speech recognition produce noisy text  i.e., text with numerous errors relative to the original printed text or speech transcript n With good retrieval model, effectiveness of search is not significantly affected by noise  Due to redundancy of text  Problems with short texts 7

8. OCR Examples 8

9. Speech Example 9

10. Images and Video n Feature extraction more difficult n Features are low-level and not as clearly associated with the semantics of the image as a text description n Typical features are related to color, texture, and shape  e.g., color histogram “Quantize” color values to define “bins” in a histogram For each pixel in the image, the bin corresponding to the color value for that pixel is incremented by one  images can be ranked relative to a query image 10

11. Color Histogram Example Peak in yellow 11

12. Texture and Shape n Texture is spatial arrangement of gray levels in the image n Shape features describe the form of object boundaries and edges n Examples. shape 12

13. Video n Video is segmented into shots or scenes  continuous sequence of visually coherent frames  boundaries detected by visual discontinuities n Video represented by key frame images  e.g., first frame in a shot 13

14. Image Annotation n Given training data, can learn a joint probability model for words and image features n Enables automatic text annotation of images  current techniques are moderately effective errors 14

15. Music n Music is even less associated with words than images n Many different representations  e.g., audio, MIDI, score n Search based on features such as spectrogram peaks, note sequences, relative pitch, etc. 15