Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, 2000 Workpackage 4 Image Analysis Algorithms Kirk Martinez, Paul Lewis and Stephen Chan Intelligence, Agents and Multimedia Department of Electronics and Computer Science University of Southampton UK

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, 2000 Task 4.1 User requirements Analysis First step was to identify the requirements of the users (note overlap with other workpackages) Required Output: Collation of scenarios and functionality Output achieved in collaboration with other participants and delivered in the form of some initial scenarios and a set of 16 goals. These are published as part of the System design document.

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, 2000 Task 4.3 Recognition Algorithm Development PM 8-24 Aim is to develop image analysis algorithms to meet user requirements Required outputs: Image content analysis software and report Consider 4.1 and 4.3 together in this presentation

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, 2000 Goals G1 Matching of similar images (includes “have you got this picture”) G2 Automatic search using synonyms G3 Search based on features oriented to the restoration framework - uv spotmeasures - x-ray and reverse pic views of frames - craquelure classifier -search based on “butterfly” supports in the frame

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, 2000 Goals Cont. G5 Access information quickly and easily G6 Search based on colour G7 Query by low quality images (especially faxes) G8 Query by sketch G9 Query refinement G10 Joint retrieval by content and by text G11 Use of publishing products built on the Artiste system G12 Detail finding

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, 2000 Goals Cont. G13 Search using multilingual vocabulary G14 Respect installation site privacy and security policy G15 Produce a sustainable system after the end of the project G16 Be consistent with partners’ predefined technical constraints

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, 2000 Analysis The goals were analysed in terms of the implications for image analysis Possible image processing (IP) approaches were identified for goals requiring IP Six distinct groups of algorithms were identified together with the goals to which they could contribute.

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, Algorithms which will find similarity matches based on global histograms (colour or grey scale) between a query image or sub-image and images in the database collections. Could contribute to goals 1,3,7 Useful for basic image matching May contribute to style search and classification Potentially faster than spatial-colour matching methods Status: Implemented at IAM

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, 2000 Example of Global Colour Histogram Search Query image Best Matches

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, Algorithms which will find similarity matches between a query image or sub-image and images or sub-images in the database using spatial colour distributions. Goals: 1,3,6,10,12 Takes into account the spatial arrangement of colours Finds similar colour patterns at similar locations Or similar colour patterns at any location Status: Implemented a hierarchical colour coherence vector based matcher in IAM

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, 2000 Example of H-CCV Matching Query sub-image Best Matches

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, Algorithms which will segment image into regions of similar texture and record feature vectors representing texture for each main regions. User can then indicate a query texture either by indicating a region in a particular image or selecting a texture from a texture palette. Images in the database containing texture regions matching the query are then retrieved. 3. Algorithms which will segment image into regions of similar texture and record feature vectors representing texture for each main regions. User can then indicate a query texture either by indicating a region in a particular image or selecting a texture from a texture palette. Images in the database containing texture regions matching the query are then retrieved. Goals: 1,3,12 Status: Previously implemented texture extraction algorithms Not yet implemented automatic texture segmentation

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, Algorithms which will match an outline query shape with similar shapes within database images. Goals: 1,3,4,7.8,10,12 Pre extracting all shapes of all objects in all images is impossible i.e. can not pre-index shapes! Techniques like the Generalised Hough Transform (GHT) use evidence accumulation to find a shape in an image and are related to template matching. They are computationally intensive Status: Not yet implemented for this project

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, Algorithms which will detect and in some cases analyse specific image features 5. Algorithms which will detect and in some cases analyse specific image features Goals: 4 May be able to use e.g. the GHT Will also require specially tailored algorithms Status: Not yet implemented

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, Algorithms to provide basic image manipulation Goals: All involving image handling Include operations like image conversion, compression, scaling, rotation etc Most are widely available but may need re-implementing or tailoring in context of Artiste Status: Partial implementation

Project IST_1999_ ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N.1: Paris, C2RMF, November 28, 2000 Environment for Algorithm Testing Test-beds in the IAM lab include VIPS and MAVIS 2 Algorithms developed as “stand alone” modules which deliver feature vectors (FVs) and modules for calculating similarity between FVs MAVIS 2 is a multimedia retrieval and navigation environment It associates media content and the concepts they represent Concept layer equivalent to a thesaurus Allows integrated content and concept based searching with query scope expansion