Content Based Image Retrieval Miguel Arevalillo-Herráez

Contents Introduction Approaches Possible extensions to 3D Information retrieval Image retrieval CBIR Approaches Combining similarity measures Full CBIR systems Possible extensions to 3D Results and Conclusions

Concepts Information retrieval Image retrieval Objects are documents Concept of a query Image retrieval Objects are images Content Based Image retrieval

Common setup for CBIR

The method How do we judge how similar two images are?

The method How do we judge how similar two images are? - feature vectors

The method How do we judge how similar two images are? - feature vectors How do we compare these vectors?

The method How do we judge how similar two images are? - feature vectors How do we compare these vectors? distance funcions defined over the feature space.

The method How do we judge how similar two images are? - feature vectors How do we compare these vectors? distance funcions defined over the feature space. How are these distances combined to yield a composite similarity value?

The method How do we judge how similar two images are? - feature vectors How do we compare these vectors? distance funcions defined over the feature space. How are these distances combined to yield a composite similarity value? Normalization and combination

The method How do we judge how similar two images are? - feature vectors How do we compare these vectors? distance funcions defined over the feature space. How are these distances combined to yield a composite similarity value? Normalization and combination How are multiple selections combined?

The method How do we judge how similar two images are? - feature vectors How do we compare these vectors? distance funcions defined over the feature space. How are these distances combined to yield a composite similarity value? Normalization and combination How are multiple selections combined? Multiple selection approaches

The method How do we judge how similar two images are? - feature vectors How do we compare these vectors? distance funcions defined over the feature space. How are these distances combined to yield a composite similarity value? Normalization and combination How are multiple selections combined? Multiple selection approaches

Normalization and Combination Rules Classical normalization rules: Gaussian Linear Classical combination rules: Sum Product Linear combination

Probabilistic Approach For each distance function we estimate the probability that the user considers that two images are similar, for every possible distance value: p(similar | di) This is performed from a training set

Probabilistic Approach For each distance function we estimate the probability that the user considers that two images are similar, for every possible distance value: p(similar | di) This is performed from a training set p(similar | d1, d2, d3,…,dn)  p(similar | d1) x p(similar | d2) x p(similar | d3) x … x p(similar | dn)

Handling Multiple Selections Classical Approaches: Query point movement and axis re-weighting Support Vector Machines Probabilistic and Regression Approaches Other interesting approaches: SOM based Nearest neighbour

Fuzzy Approach - Concepts Need to deal with uncertainty of the data Classical set: Elements are or are not in the set Fuzzy set: Elements have a degree of membership to the set

Fuzzy approach Assumes an underlying search model Any image of interest should be perceptually similar to each of the pictures in the set Positive in at least kpos characteristics. Any image of interest should be perceptually different from each of the pictures in the set Negative in at least kneg characteristics.

Fuzzy approach Every iteration the user is more exigent: Kpos and Kneg vary at each iteration

Fuzzy Approach

Genetic Approach An evolutionary algorithm attempts to solve a problem applying Darwin’s basic principles of evolution on a population of trial solutions to a problem, called individuals.

Genetic Approach

Genetic Approach Key issues: Existence of fitness function Relevance feedback defines population and fitness Maintaining consistency How do we judge next generation?

Genetic Approach

Genetic Approach

Possible extensions to 3D How do we judge how similar two images are? - feature vectors How do we compare these vectors? distance funcions defined over the feature space. How are these distances combined to yield a composite similarity value? Normalization and combination How are multiple selections combined? Multiple selection approaches

Results and Conclusions Introduction to the CBIR problem Feature extraction Definition of distance funcions normalization and combination Handling multiple selections Posible extensions to 3D