1. A Graphical User Interface for a Fine-Art Painting Image Retrieval System October 15-16, 2004 October 15-16, 2004

2. Multimedia Information Retrieval 2004 Introduction Students of art history learn three primary skills: Formal analysis Formal analysis Comparison Comparison Classification Classification How can computer science contribute to the development of these skills? of these skills? Girl with a Pearl Earring, Jan Vermeer, 1665

3. Multimedia Information Retrieval 2004 Working Hypothesis An Interactive Indexing and Image Retrieval System (IIR) for fine-art paintings can aid students in these endeavors by providing: An Interactive Indexing and Image Retrieval System (IIR) for fine-art paintings can aid students in these endeavors by providing: a mathematical summarization of an image a mathematical summarization of an image a measurable basis for comparing two images a measurable basis for comparing two images an elementary way to classify an image relative to those in a database an elementary way to classify an image relative to those in a database

4. Multimedia Information Retrieval 2004 Previous Work We synthesize the goals of two research areas: Classification of paintings which often requires special images (brush stroke detection) or features with little semantic relevance to art students Classification of paintings which often requires special images (brush stroke detection) or features with little semantic relevance to art students Image retrieval which aims to bridge the semantic gap Image retrieval which aims to bridge the semantic gap Can we find a feature set that satisfies the objectives of both areas while providing analytically relevant data to students?

5. Multimedia Information Retrieval 2004 System Overview The system consists of two major components: Image Database Image Database stores images, thumbnail images, and extracted features for later retrieval and analysis. stores images, thumbnail images, and extracted features for later retrieval and analysis. Graphical User Interface Graphical User Interface provides interactive query capabilities to the end user provides interactive query capabilities to the end user

6. Multimedia Information Retrieval 2004 Database Construction An XML index file stores extracted features and control information An XML index file stores extracted features and control information A file system stores images and thumbnail images A file system stores images and thumbnail images

7. Multimedia Information Retrieval 2004 Database Construction – Cont. XML Index File File System

8. Multimedia Information Retrieval 2004 Global Feature Extraction Two different kinds of features are extracted: Palette features Palette features concern the set of colors in an image (color map) concern the set of colors in an image (color map) examples: palette scope examples: palette scope Canvas features Canvas features concern the spatial and frequency distribution of colors in an image (image index) concern the spatial and frequency distribution of colors in an image (image index) examples: max, min, median, mean (for each color channel) examples: max, min, median, mean (for each color channel)

9. Multimedia Information Retrieval 2004 Example: Palette Scope Palette Scope -- the total number of unique colors used in an image. We expect Dali’s piece to have a higher palette depth than Mondrian’s work. Hallucinogenic Toreador Salvador Dali, 1970 Composition with Large Blue Plane, Red, Black, Yellow, and Gray Piet Mondrian, 1921

10. Multimedia Information Retrieval 2004 Example: Palette Scope – Cont. Artist RGB Raw Palette Scope Normalized Mondrian300050.00178843 Dali766130.00456649 We see that Dali uses twice as much of the color spectrum as Mondrian. Palette scope is an important feature for artist and period style identification because many styles are defined by color, i.e. Picasso’s Blue Period and fauvism.

11. Multimedia Information Retrieval 2004 Graphical User Interface

12. Multimedia Information Retrieval 2004 Test Results Two types of tests were conducted: Feature tests Feature tests Interactive tests Interactive tests

13. Multimedia Information Retrieval 2004 Test Results – Cont. Training Set Test Set Percent Correct 363694 20020088 20020083 Les Demoiselles d’Avignon, Pablo Picasso, 1907. Road with Cypress and Star, Vincent Van Gogh, 1890. Feature test to distinguish the work of Picasso and Van Gogh.

14. Multimedia Information Retrieval 2004 Initial Interactive Test Database of 10 works of each of the following ten artists: Braque, Cezanne, De Chirico, El Greco, Gauguin, Modigliani, Mondrian, Picasso, Rembrandt, and Van Gogh. Training Set Testing Set Percent Correct 1009081

15. Multimedia Information Retrieval 2004 Interactive Test on Web Museum Database Artist Training Set QueriesSuccessPercent Aertsen98787.5 El Greco 108450.0 Hopper108112.5 Malevich1011654.5 Monet1010660.0 Morisot1011545.5 Rembrandt10322371.9 Renoir10381231.6 Turner1010330.0 Velazquez108787.5 Overall50029914749.2

16. Multimedia Information Retrieval 2004 Evaluation Test Results Evaluation of Web Museum Test Results Overall result: 49.2% accuracy Overall result: 49.2% accuracy 29.2% better than blind guessing (10 guesses/50 artists = 20%) 29.2% better than blind guessing (10 guesses/50 artists = 20%) Dissecting the classification mistakes reveals some intelligent mistakes Dissecting the classification mistakes reveals some intelligent mistakes Rembrandt is most often confused with Caravaggio, Ast, and Vermeer Rembrandt is most often confused with Caravaggio, Ast, and Vermeer

17. Multimedia Information Retrieval 2004 Conclusions Simple palette and canvas features are sufficient for an interactive classification system Simple palette and canvas features are sufficient for an interactive classification system A single feature set can serve for classification and image retrieval applications A single feature set can serve for classification and image retrieval applications A general feature set can adequately serve for educational applications A general feature set can adequately serve for educational applications Although showing promise, we currently have a low confidence system Although showing promise, we currently have a low confidence system

18. Multimedia Information Retrieval 2004 Future Work Can computer science provide an empirical framework for the study of painting? Can computer science provide an empirical framework for the study of painting? Quantitative description Quantitative description Falsifiable statements Falsifiable statements Hypothesis verification Hypothesis verification