Workshop on Preserving Intellectual Assets: Institutional Repositories and Open Access TEI Thessalonikis, Sindos, September 2006 An introduction to image retrieval Professor Dick Hartley Manchester Metropolitan University

Introduction to image retrieval Why is image retrieval important for digital libraries and institutional repositories? Why is image retrieval difficult? What are the approaches to image retrieval?

How are we going to achieve this? The bad news –I am going to do some talking –So, you are going to do some listening (I hope!!!!) The good news –You are going to do some work ! (well it is a workshop!!!!)

How are we going to achieve this? Day 1 –Why is image retrieval important? –Why is it difficult? –One approach to image retrieval –Practical image retrieval exercise Day 2 –A second approach to image retrieval –Practical exercise in image indexing –Research on image seeking behaviour

What do I mean by image retrieval? Digitized images of text Digital images in every conceivable subject from medical imaging, through satellite imagery to art history

Why is image retrieval important? Image information is crucial in many contexts Huge quantities of image data is now available in digital form Digital information on every imaginable subject is readily available on the Web Many digital libraries contain digital information; this is pointless unless it can be effectively retrieved

So…. Research and practical developments in image retrieval and in understanding of image seeking behaviour have been major areas of development in information retrieval in the last decade

Why is it difficult? What is an image about? Look at the following examples……

Are you Offended?

Why is it difficult? I want a picture of a tower in Greece at night I want a picture of a bridge in Edinburgh during the summer

Approaches to image retrieval Content-based image retrieval Concept-based image retrieval

Content-Based Image Retrieval Semi-automatic or automatic extraction, indexing and retrieval of images by their visual attributes.

Similarity Measures City-Block Distance Minkowsky Distance

Retrieval by Colour (Global) Similar Colour (Global)

False Positives

Retrieval by Colour (Local) Dissimilar Local Colour Colour Histogram Intersection

Shape Retrieval

Inference

Noise

Trademark Image Retrieval

Device Marks

Texture

Image Query Paradigms Relational-Based –SQL [Codd, 1970]. –ISQL [Assmann, Venema, and Hohne, 1986], –PROBE [Orenstein, and Manola, 1988], –PSQL [Roussopoulos, Faloutsos, and Sellis, 1988] –Spatial SQL [Egenhofer, 1991]. SELECT city, state, population, location FROM cities ON us-map WHERE location within (4+4, 11+9) AND population > 450,000 Tabular-based data model –Query By Example (QBE) [Zloof, 1977]. Aggregate by Example [Klug, 1981] Generalised Query by Example [Jacobs and Walczak, 1983] Office by Example [Whang et al. 1987] Time by Example [Tansel et al., 1989] Natural Forms Query Language (NFQL) [Embley, 1989]. Query by Pictorial Example (QPE) [Chang and Fu, 1980]. PICQUERY [Joseph and Cardenas, 1988].

Query by Visual Example

QBIC Niblack et al., Lee et al., Bird et al., Bird et al., “One of the key challenges that remains in making the technology pervasive and useful is the design of the user interface.” [Flickner et al., 1997.]

Query by Image

Query by Icon

Query by Paint

Query by Sketch Query by Visual Example

Beyond Query by Visual Example 2D/3D Visualization –ImageVIBE [Cinque et al., 1998]. –3DVIBE [Santini and Jain, 1997]. Virtual Reality –Query by Photograph [Assfalg et al., 2000]. Taxonomy Human Perception of Images –Burford et al. [2002].

CBIR Summarized CBIR permits retrieval by image attributes –Colour, shape, texture (or a combination)

CBIR Advantages No metadata necessary Possible to “index” a huge volume of material and rapidly Does not depend on interpretation of meaning

CBIR disadvantages “Semantic gap” between what users want and what CBIR systems can achieve No sensible means by which queries can be presented to a CBIR system

CBIR uses Restricted areas such as trade mark searching

Time for a break, then you can experiment with an operational CBIR system