1. A Lightweight Image Retrieval System for Paintings T. Lombardi, S. Cha, and C. Tappert January 19th, 2005 January 19th, 2005

2. Electronic Imaging 2005 Introduction Students of art history learn three primary skills: Formal analysis Formal analysis Comparison Comparison Classification Classification How can computer science contribute to the development of these skills? of these skills? Figure 1: Girl with a Pearl Earring, Jan Vermeer, 1665

3. Electronic Imaging 2005 Working Hypothesis An Interactive Indexing and Image Retrieval System (IIR) for fine-art paintings can aid students in these endeavors by providing: An Interactive Indexing and Image Retrieval System (IIR) for fine-art paintings can aid students in these endeavors by providing: a mathematical summarization of an image a mathematical summarization of an image a measurable basis for comparing two images a measurable basis for comparing two images an elementary way to classify an image relative to those in a database an elementary way to classify an image relative to those in a database

4. Electronic Imaging 2005 Previous Work We synthesize the goals of two research areas: Classification of paintings: Classification of paintings: R. Sablatnig, P. Kammerer, and E. Zolda, “Hierarchical Classification of Paintings Using Face- and Brush Stroke Models”, in Proc. of the 14th International Conference on Pattern Recognition (1998). D. Keren, “Painter Identification Using Local Features and Naïve Bayes”, in Proc. of the 16th International Conference on Pattern Recognition (2002). Image retrieval which aims to bridge the semantic gap: Image retrieval which aims to bridge the semantic gap: J. Corridoni, A. Del Bimbo, and P. Pala, “Retrieval of Paintings using Effects Induced by Color Features”, in Proc. of the International Workshop on Content-Based Access of Image and Video Databases (1998). Can we construct a feature set that satisfies the objectives of both areas while providing analytically relevant data to students?

5. Electronic Imaging 2005 System Overview The system consists of two major components: Image Database Image Database stores images, thumbnail images, and extracted features for later retrieval and analysis. stores images, thumbnail images, and extracted features for later retrieval and analysis. Graphical User Interface Graphical User Interface provides interactive query capabilities to the end user provides interactive query capabilities to the end user

6. Electronic Imaging 2005 Database Construction An XML index file stores extracted features and control information. An XML index file stores extracted features and control information. A file system stores images and thumbnail images. A file system stores images and thumbnail images. The open design of the database contributes to the goals of ease of use and exchange of information. The open design of the database contributes to the goals of ease of use and exchange of information.

7. Electronic Imaging 2005 Database Construction – Cont. Figure 2: XML Index File Figure 3: File System

8. Electronic Imaging 2005 Global Feature Extraction Two different kinds of features are extracted: Palette features Palette features concern the set of colors in an image (color map) concern the set of colors in an image (color map) examples: palette scope examples: palette scope Canvas features Canvas features concern the spatial and frequency distribution of colors in an image (image index) concern the spatial and frequency distribution of colors in an image (image index) examples: max, min, median, mean (for each color channel) examples: max, min, median, mean (for each color channel)

9. Electronic Imaging 2005 Sample Feature Set Feature Name Description and Notes Max Max value of H, S, and V channels Min Min value of H, S, and V channels Mean Mean of H, S, and V channels Median Median of H, S, and V channels Standard Dev. Std of H, S, and V channels Color Entropy Measures the frequency distribution of color Line Count Normalized number of detected edges – Sobel edge detector Intensity Mean Arithmetic mean of values in a grayscale image Table 1: Sample Features used for Web Museum Interactive Test

10. Electronic Imaging 2005 Example: Palette Scope Palette Scope -- the total number of unique colors used in an image. We expect Dali’s piece to have a higher palette depth than Mondrian’s work. Figure 4: Hallucinogenic Toreador Salvador Dali, 1970 Figure 5: Composition with Large Blue Plane, Red, Black, Yellow, and Gray Piet Mondrian, 1921

11. Electronic Imaging 2005 Example: Palette Scope – Cont. Formal definition of Palette Scope (U): U = C/P Where C=Total # of unique colors measured in RGB or HSV triples. P= Total # of pixels in an image.

12. Electronic Imaging 2005 Example: Palette Scope – Cont. Artist Total Pixels (P) Total Colors (C) Palette Depth (U) Mondrian35970022420.00623 Dali16577538990.02351 Table 2: Palette Scope statistics. We see that Dali uses more of the color spectrum than Mondrian. Palette depth is an important feature for artist and period style identification because many styles are defined by color, i.e. Picasso’s Blue Period and fauvism.

13. Electronic Imaging 2005 Graphical User Interface The GUI consists of three primary windows for: The GUI consists of three primary windows for: Analysis Analysis Comparison Comparison Classification Classification

14. Electronic Imaging 2005 Analysis Window Figure 6: The Analysis Window

15. Electronic Imaging 2005 Comparison Window Figure 7: The Comparison Window

16. Electronic Imaging 2005 Classification Window Figure 8: The Classification Window

17. Electronic Imaging 2005 Test Results Two types of tests were conducted: Feature tests Feature tests Feature tests focus on the accuracy of specific collections of features. Feature tests focus on the accuracy of specific collections of features. Interactive tests Interactive tests Interactive tests assess the accuracy of the system as a whole. Interactive tests assess the accuracy of the system as a whole.

18. Electronic Imaging 2005 Feature Test Training Set Test Set Percent Correct 363694 20020088 20020083 Figure 9: Les Demoiselles d’Avignon, Pablo Picasso, 1907. Figure 10: Road with Cypress and Star, Vincent Van Gogh, 1890. Table 3: Feature test to distinguish the work of Picasso and Van Gogh.

19. Electronic Imaging 2005 Initial Interactive Test Database of 10 works of each of the following ten artists: Braque, Cezanne, De Chirico, El Greco, Gauguin, Modigliani, Mondrian, Picasso, Rembrandt, and Van Gogh. Training Set Testing Set Percent Correct 1009081 Table 4: Initial Interactive Test

20. Electronic Imaging 2005 Interactive Test: Web Museum Artist Training Set QueriesSuccessPercent Aertsen99555.6 El Greco 107457.1 Hopper107342.9 Malevich1011872.7 Monet1010660.0 Morisot1011763.6 Rembrandt10332575.8 Renoir10381436.8 Turner1010440.0 Velazquez1088100.0 Overall50029316556.3 Table 5: Results from Web Museum Interactive Test

21. Electronic Imaging 2005 Evaluation Test Results Evaluation of Web Museum Test Results Overall result: 56.3% accuracy Overall result: 56.3% accuracy 36.3% better than blind guessing (10 guesses/50 artists = 20%) 36.3% better than blind guessing (10 guesses/50 artists = 20%) Dissecting the classification mistakes reveals some intelligent mistakes Dissecting the classification mistakes reveals some intelligent mistakes Rembrandt is most often confused with Caravaggio, Ast, and Vermeer Rembrandt is most often confused with Caravaggio, Ast, and Vermeer

22. Electronic Imaging 2005 Conclusions Simple palette and canvas features are sufficient for an interactive classification system Simple palette and canvas features are sufficient for an interactive classification system A single feature set can serve for classification and image retrieval applications A single feature set can serve for classification and image retrieval applications A general feature set can adequately serve for educational applications A general feature set can adequately serve for educational applications Although showing promise, we currently have a low confidence system Although showing promise, we currently have a low confidence system

23. Electronic Imaging 2005 Future Work Add texture features Add texture features Improved color features: hue histograms Improved color features: hue histograms Improved distance metrics: modulo comparison of hue histograms Improved distance metrics: modulo comparison of hue histograms Test against larger datasets Test against larger datasets