Analysis and classification of images based on focus Quinton Smith Mentored by Dr. David Doria FlickrTM, 500pxTM, YogileTM and ZenfolioTM. These odd terms have one thing in common; they are all sites designed for photo sharing by photographers. Within each site there is a large variety of different photos - from portraits to landscapes. To allow users to find what they are looking for, these photos must be categorized. But how are these images placed in these categories? Typically the user manually assigns a label to each image. Automating this process would expedite this manual process, along with ensuring consistency in the categorization between photographers. To automate the categorization, features in the images must be quantified so that an algorithm can distinguish between classes of images using machine learning (Han, Qi, 2005). This has been done with details like texture (Haralick, Shanmugan, Dimstein, 1973). In this work, it is proposed that the use of analysis of the regions of the image that are in focus to classify images. In the majority of landscape photos, the photographer tries to keep the whole scene in focus so that nothing is blurry, while portraits tend to focus on the eyes of the intended target and leave the rest of the image out of focus. Introduction Methods and Materials (cont.) focused area of an image. To compare images, values derived from focus, such as the percent of focused pixels, and the number of blobs of focus pixels, can be compared between images. In order to study these regions of focus, techniques were drawn from morphology – the study of shape. The shape of the blobs of pixels were studied and changed. It can be used to increase the size of blobs so that small, one pixel gaps between blobs, which can be considered connected, are automatically filled. A process like this is called a morphological dilation, similar to when one’s pupil dilates, it becomes larger. Opposite to dilations, morphology can also be used to shrink a blob so that it no longer connects to other blobs. This is called a morphological erosion and is useful for cutting of unimportant pixels from the focused area. If there is a small bridge between two blobs that should not be connected, then a process called a morphological closing is performed. A closing allows small, one to two pixel thick bridges between blobs to be cut without actually decreasing the size of the blobs. Application of morphology created the focus maps in Figure 2. Figure 2: The progression of the code as it finds the most focused part of an image, from the original, left, to the focus blobs, middle, to the most focused blob in the image, right, for a landscape (Golightly, 2014), top, and a portrait, bottom. After determining the most focused area of the image, the next step was to collect data and build a classifier. The size of the largest in-focus region, as well as the total number of focus blobs that were present in the image were obtained. This data was collected for 20 training images, 10 from each category of portrait and landscape. Averages were found, and used to categorize a set of 100 test images, 50 from each category. By creating a linear discriminant line between the two average points, and plotting the test points on the same graph, one can visually see the category that each image would be placed in. The algorithm did this by finding the geometric distance (Graph 1) between a test image’s data points, and the averages of each category. The shortest distance was considered the image’s category. Results Graph 2: These two graphs show the number of correctly categorized images versus those incorrectly categorized for the two categories. The landscape photos had greater success at being categorized. Graph 1: Scatterplot containing points consisting of each image’s area and number of blobs, as well as the average for each image type. The closer a point is to the average, the smaller the geometric mean. Points to the left of the black line were considered portraits, those on the right were landscapes. An image’s focus is the area of the image that is “sharp” as compared to the rest of the image. What this means is that this area has the most detail in the image, and is not blurry in any way. The differences in the colors of the surrounding pixels can be considered to be detail, and thus could be used to determine the level of focus of a pixel relative to its surroundings. The variance, or spread, of the pixel values is a good estimate of the amount of detail at a point in an image. Using the Python programming language, and the built-in libraries SciPy and NumPy, the variance of each pixel with its surrounding pixels was found and used to create a new image, a map of the most focused areas of the image (Figure 1). This map can be used to find the most detailed and thus most Figure 1: Original images and the focus maps for a landscape, top, and a portrait, bottom. Methods and Materials Of the 100 images in our experiment, 71 of them were correctly categorized. This indicates that focus is a feasible feature to use to categorize images automatically. A major discrepancy is that the Landscape category had a 98% success rate, while the Portrait category had a 44% success rate (Graph 2). This could have been due to the wide variety of images that are present in the category of Portrait. Using additional features computed from the focus values would be beneficial. Additionally, using larger sets of training data for the learning phase of the algorithm would certainly improve the robustness of the system. Conclusions References Golightly, C. (Photographer). (2014, November 14). Quiraing Tree, Isle of Skye. [digital image]. Retrieved from https://www.flickr.com/photos/cgolightly/16490546877/ Han, Y., & Qi, X. (2005). Machine-learning-based image categorization. In Image Analysis and Recognition (pp. 585-592). Springer Berlin Heidelberg. Haralick, R. M., Shanmugam, K., & Dinstein, I. H. (1973). Textural features for image classification. Systems, Man and Cybernetics, IEEE Transactions on, (6), 610-621. Acknowledgements I owe my success to my mentor, Dr. David Doria, for his continued support and guidance through the entirety of this project. I also thank my faculty advisor, Mr. Davis, as well as Mr. Evans.