Image Retrieval Part I (Introduction)

2 Image Understanding Functions Image indexing similarity matching image retrieval (content-based method)

3 Images containing similar colors

4 Images containing similar content

5 Image Database

6

7 Variety of Similarity Similar color distribution Similar texture pattern Similar shape/pattern Similar real content Degree of difficulty Histogram matching Texture analysis Image Segmentation, Pattern recognition Life-time goal :-)

8 Image Indexing

9

10 Query by Example I

11 Query by Example 2 “Find all paintings of this shape”

12 Retrieval by Sketch PictureFider search engine

13 QBIC - Query By Image Content First and best known commercial image retrieval system IBM Almaden Research Center Commercial product Search by shape, color, texture, keyword Queries can be based on: example images, user-constructed sketches and drawings, selected color and texture patterns User can: Select colors and color distributions from a color wheel, Select textures from a predetermined selection, Adjust relative weights among shape features

14 Content-based image retrieval simple features are not semantic entities as opposed to words in a text not easy to extract semantic content choice of indexing elements is unclear low-level features color, texture, shapes high-level features a person on bicycle, a windsurfer, vase abstract features triumph, sadness, happiness

15

16 Content of Image Image = array of pixel intensities b/w: 1 bit gray: 8 bits colour: 24 bits code: index in colour lookup table

17 Image Retrieval by Intensity pair-wise comparison histogram methods central moments

18 Pair-wise comparison feature vector = raw array of pixel intensities needs model of pixel intensity distance d C compared pictured must be same size considers spatial arrangement not concise

19 Histogram methods quantise intensity space into N bins H(i) is pixel proportion with intensity of bin i distance of images I and Q (query example): histogram intersection simple, concise method global feature: locality is lost Exercise: why must the histograms be normalized?

20 Histogram Encoding

21 Center Image Histogram

22 Window Histogram Encoding

23 Window + center encoding

24 Intensity central moments instead of a histogram compute three statistical numbers per image (or window) mean (average gray level) variance (squared standard deviation) 3 rd central moment (skewness of the grey level distribution)

25 Intensity central moments mean, variance and 3 rd central moment:

26 Intensity central moments in order to have numbers which are comparable in size for each moment we use the corresponding root we get a 3-dim feature vector:

27 Moments + spatial coherence moments are a global feature spatial coherence can be added by windows centering windows+centering

28 summary (retrieval by intensities) paiwise comparison histograms central moments balance in locality through windows, centering, windows+centering

29 Retrieval by color RGB color model color histograms color moments clustering in color space

30 The RGB color model mixture of intensities of red, green and blue for active devices such as monitors some examples

31 color histograms simple methods: three 1-d histograms; one 3d histogram create a 1d histogram for each color component each: eg, one for R, one for G, one for B divide 3d color space into 3d bins for histograms: R GB

32 Color moments feature vector: 9 numbers per image or window 3 moments per color channel:

33 Clustering in color space cluster the pixels in 3d color space features: centroids and population of clusters similarity measure compares color centriods and population must take different cluster number into account

34 Summary (retrieval by color) RGB color model (or HSV color model) two color histograms moments of color clustering in color space

35 Texture perceptual phenomenon local region quality, not point quality depends on the scale texture = repeated pattern local variations in image intensity too fine as to be considered as own objects Texture feature Co-occurence matrix Gabor Wavlet Filter coeffiecients

36 Texture Examples

37 Segmentation of Images texture analysis needs segmentation of the still image to be meaningful separate objects ignore background assign texture feature per object object segmentation is a difficult task; edge detection can be useful for this

38 Texture+Color Segmentation

39 Shape Analysis Shape= class of geometric objects invariant under information preserving description (for compression) non-information preserving (for retrieval) boundary based (ignore interior) region based (boundary+interior) Shape features Chaincodes (boundary based) Fourier Descriptor (boundary based) ect

40 Examples of shapes

41 Chaincodes contour description by a string of directions Freeman chain code f 1 f 2 f 3... eight possible directions for each pixel translation invariant, not rotation invariant

42 Chain code histograms used for distance measure same histogram represents many shapes percentage histogram is scale invariant

43 Retrieval by shape

44 Retrieval by shape

45 Content-Based Image Retrieval many different low-level features can be computed, why did we do this? to describe a complex multimedia object to automatically annotate mm objects with salient properties (eg, color shape, texture) to be able to compute similarities of mm objects to make mm objects searchable by content high-level feature are not (yet) accessible, but research attempts to construct high-level concepts from low-level features

46 Content-Based Image Retrieval Which features are best for searching? Depends on the information need: looking for sunset holiday picture in your digital shoebox? Use color histograms want to build a wallpaper customer database? Use color+texture want to build a b/w sketch database for technical industrial designs? Use shape descriptors Not sure which features are best for a query (eg, if you also have abstract features such as Fourier coefficients)? Deploy relevance feedback and let the system learn the relevant feature for this query...

47 Content-based Image Retrieval Summary: we looked at retrieval by intensity color models retrieval by color retrieval by shape retrieval by texture these features can be used in a general framework for content-based retrieval as discussed earlier

48 Experiment: Setting up CBIR search engine Create directory c:/Experiments/database copy database file “texturedatabase” into the directory copy the following two subdirectory into c:/Experiments/database ImageDatabase ShapeColor Open database (“java COM.cloudscape.tools.cview”) install the database “texturedatabase” Close database

49 Compile file.Java & deploy application open cmd and type the following command javac *.java run J2ee server run deploytool deploy your application to include the following Web Components: index.jsp MyLocalesRbf2 MyDateJose2 and all the jpeg images stored in the directory called “ImageDatabase”.

50 Summary of Fundamental CBIR Content-based image retrieval (CBIR) CBIR Algorithm: First step --- Image Indexing Second step --- content matching