Integrating Color And Spatial Information for CBIR NTUT CSIE D.W. Lin

References  L. Cinque, G. Ciocca, S. Levialdi, A. Pellicano, and R. Schettini, “Color-based image retrieval using spatial- chromatic histograms,” Image and Vision Computing, 19 (2001)  M.S. Kankanhalli, B.M. Mehtre, and H.Y. Huang, “Color and spatial feature for content-based image retrieval,” Pattern Recognition Letters, 20 (1999)  S. Berretti, A.D. Bimbo, and E. Vicario, “Spatial arrangement of color in retrieval by visual similarity,” Pattern Recognition 35(2002)

The representation of color  Color histogram – Global color histogram – Global color histogram + spatial info. – (fixed) Partition + local color histogram  Dominant color – Extracting the representative colors of image via VQ or clustering (e.g. k-means algorithm) – Spatial info. can be attained Histogram refinement Specific-color pixel distribution (single, pair, triple …) Edge histogram … Non-adaptive

Spatial-chromatic histograms [1]  SCH – global color histogram with info. about (single) pixel distribution  SCH attempts to answer: – How many meaningful colors?  color space quantization – Where the pixels having the same color?  location of region(with same color) – How are these pixels spatially arranged?  distribution of region

SCH – Color representation  Color representation – CIELAB  Munsell  ISCC-NBS  CIELAB – CIEXYZ CIELAB CIELUV – a, b: opponent color ( green  red, blue  yellow )

SCH – Color representation  Munsell color system – Hue value chroma

SCH – Color representation  ISCC-NBS Centroid Color System – Partitioning the Munsell color system into 267 blocks, each blocks represented by an unique linguistic tag and the block centroid (Munsell coordinates)  Using back-propagation NN to transform – CIELAB  Munsell  ISCC-NBS

SCH – Feature vector  The definition of SCH for image I S I (k) = (h I (k), b I (k), σ I (k)) – k: k th quantized color (1~c) – h I (k): pixel amount(ratio) – b I (k): baricenter (normalized mean coordinates) – σ I (k): standard deviation of (spread)  Properties: – Insensitive to scale changes(via normalization) – Compact representation and rapidly computing

SCH – Similarity measure  Similarity function – c: number of quantized color – d(·): Euclidean distance – : max. distance

SCH – Effectiveness measure – S: relevant items in DB – : retrieved set (short list) for a query – : relevant items in retrieved set

Color and spatial clustering [2]  k-means algorithm – Iteration version – Two-pass version – VQ (LBG algorithm)  Proposed color clustering (two-pass) – Generating a new cluster while d(p, C i ) > T – Merging those clusters with small population to the nearest cluster

Color and spatial clustering  Spatial clustering – Based on the clustered color layer – Using connected components labeling to separate the spatial clusters – Discarding those clusters with small population or lower density(embedded rectangular)

Feature vectors  For image I, color clusters can be given C ci = {R i, G i, B i, λ ci, x ci, y ci } i: 1..m (number of color cluster) R i, G i, B i : representative color of cluster λ ci : pixel ratio of cluster to total x ci, y ci : centroid of cluster f c ={C ci |i=1, 2, …, m}  Do the same to color-spatial clusters

Similarity measure –  1 : color distance between color cluster(RGB) –  2 : relative frequency of pixels of color cluster ( ) –  3 : spatial distance between color cluster(x, y) –  4 : relative frequency of pixels of color-spatial cluster ( ) –  5 : spatial distance between color-spatial cluster(x, y)

Spatial arrangement of color[3]  The back-projection from dominant colors to the image results in an exceedingly complex model(e.g. [2])  Authors proposed a descriptor, called weighted walkthroughs, to capture the binary directional relationship of two complex sets of pixels

Weighted walkthroughs – The model can be extended to represent the relationship of two sets A, B – w 11 evaluates the number of pixel pairs a  A and b  B such that b is upper right from a +,+ w +1+1 a B +, - -, + -, - C i : characteristic function of negative/positive real number |B| : area of Bi,j : ±1

Compositional computation  Reducing the region to a set of rectangular  The weight between A and B 1  B 2 can be derived by linear combination of A/B 1 and A/B 2

Distance of WW  3 directional indexes: – w H (A, B) = w 1, 1 (A, B) + w 1, -1 (A, B) – w V (A, B) = w -1, 1 (A, B) + w 1, -1 (A, B) – w D (A, B) = w -1, -1 (A, B) + w 1, 1 (A, B)  Spatial distance A B A B +,+ w +1+1 a B +, - -, + -, - A B

Arrangements comparing  Image model: E: set of spatial entities (color-clustered region) a: E  A  { any a } (chromatic label) w: E  E  W  { any s } (spatial description)

Arrangements comparing  Distance between image model Q and D –  : injective function(interpretation, association between query and model image) – D A : chromatic distance (L*u*v*) – D S : spatial distance – N q : number of entities in query

Future works  Finish the color-spatial study(geometric- enhanced histogram)  Study wavelet and JPEG 2000