T.Sharon 1 Internet Resources Discovery (IRD) Introduction to MMIR

2 T.Sharon Contents Visual Information Retrieval (VIR) –Images –Video Video Information Retrieval (VIR) Music Information Retrieval (MIR)

3 T.Sharon Visual Information Retrieval Introduction VIR system VIR information domains Querying video Advanced topics

4 T.Sharon Introduction What is VIR? Who needs it? Questions and problems

5 T.Sharon What is VIR? Query VIR System VIR allows users to query and retrieve visual information. Queries will be done according to information content.

6 T.Sharon Who needs VIR? Libraries Museums Scientific Archives Image Warehouses

7 T.Sharon Questions and Problems How can we search visual information? How can visual and non-visual information can be searched together? Problems: –visual information is subjectively interpreted. –few representations: images, graphics, video, animations, stereoscopic images. –requires substantial amount of resources. Dog??

8 T.Sharon VIR System Architecture Query formulation Match and ranking Query answer Refinement (relevance feedback)

9 T.Sharon Architecture of VIR System

10 T.Sharon Query Formulation Query by example: –sketch an example –give an image Query by giving values to visual features: –% colors –texture –describe textually but use visual tools to define values.

11 T.Sharon Query matching and ranking Similarity test Using combination of features, for example: –colors –texture –shape –motion –additional information Actions in feature space can be: –maximal distance –K nearest neighbors Feature 1 Feature 2 Feature 3

12 T.Sharon Query Answer  Thumbnails: Images –DC images Video –built from selected DC images (key frames)

13 T.Sharon Query Refinement Using a result image from previous query. Launch a new query. Modifying a result image with an image processing tool to specify an an additional criteria. Changing relative weights of visual features and get a new ranking to the previous results.

14 T.Sharon VIR Information Domains Information domain Queries at Pixels Level –examples –problems Implementations –color –color complex –shape

15 T.Sharon Information Domains Metadata information –alphanumeric, database scheme Visual characteristic –contained in the object –achieved by using computational process, usually image processing

16 T.Sharon Queries at Pixels Level - Examples  Find all objects for which the 100th to 200th pixels are orange (RGB=255,130,0).  Find all the images that have about the same color (certain RGB) in the central region (relative or absolute).  Find all images that are a shifted version of this particular image, in which the maximum allowable shift is D.

17 T.Sharon Queries at Pixels Level -Problems  Pixel queries are noise sensitive  couple of noise pixels can cause to discard a good image.  Do not work on rotations.  Changes in lightning and imaging conditions effect pixels significantly and bias queries.

18 T.Sharon Implementations Pixels location combined with Database scheme built by humans Example techniques: –Color –Color complex –Texture –Shape

19 T.Sharon Color Method: –Color definition Hue (color spectrum) Saturation (gray) –Calculate histograms Enables queries: –Find all images for which more than 30% is sky blue and more than 25% is grass green –Sort histogram drawers, find 5 most frequent colors, find all other images with these color features –Find all images far from this image only D

20 T.Sharon Color Complex Method: Create histograms quad-tree: Calculate image histogram Divide image to quarters and calculate histogram for each quarter Continue recursively till 16x16 squares Enables queries: –Find images for which: more than 20% red-orange pixels in the right upper quarter more than 20% yellow pixels in the left upper quarter about 30% brown pixels in the bottom half of the picture –Find all images with red patch in the middle and blue patch around.

21 T.Sharon Shape Method: –Suppose we have graphics collection (clip arts) contain pure colors (little hue changes, no saturation) –Divide image to color areas so that each area contains pixels with the same pure color –Calculate features for each area: color, area, elongation (sqrt(perimeter)/area), centrality (distance of shape centroid to image center) Enables queries: –Find all images containing white squares in the center –Find all images containing 2 blue circles close to the center

22 T.Sharon Examples: Existing Systems SaFe Virage QBIC (stamps) download! (photos) MetaSEEK bin/MetaSEEk_cate bin/MetaSEEk_cate WebSEEK VisualSEEK MELDEX bin/gw?c=meldex&a=page&p=coltitle bin/gw?c=meldex&a=page&p=coltitle

23 T.Sharon SaFe

24 T.Sharon QBIC - Histogram Query

25 T.Sharon QBIC - Color Layout Query