Creating Better Thumbnails Chris Waclawik

Project Motivation Thumbnails used to quickly select a specific a specific image from a set (when lacking appropriate metadata), but… Larger sets require smaller, less detailed thumbnails.

Thumbnail effectiveness Can you find the picture with the dog in it?

Thumbnail effectiveness Can you find the picture with the dog in it?

Thumbnail effectiveness Can you find the picture with the dog in it?

Problem Statement How can thumbnails be made more effective?

Show only the salient parts

Two subproblems How do we determine the salient portions of the image? Knowing the above, how should we crop the image?

Determining Image Saliency F. Liu, M. Gleicher, Region enhanced scale- invariant saliency detection, in: Proceedings of IEEE International Conference on Multimedia & Expo (ICME), 2006, pp Two-part method: – Scale-invariance makes this method suitable for determining salient features of many sizes. – Region-enhancement aids in making logical crops later on.

Scale-invariant Saliency Image transformed to perceptually uniform color space (Lu*v*). – Difference in color values proportional to perceived difference. Gaussian pyramid created and a contrast map is calculated for each level. – The contrast of each pixel is the sum of its distances from its neighbors (in Lu*v* space). – Pixels closer to the center are given more weight. The contrast maps are summed to find scale invariant saliency. Algorithm implemented in Matlab.

Region-enhanced Saliency Regions extracted from the image using any preferred method. – Mean shift method used in paper (and in this project). – C++ code from EDISON, Matlab wrapper function from shawnlankton.com. Scale-invariant saliency is averaged over each region.

Saliency-based Cropping B. Suh, H. Ling, B. B. Bederson, and D. W. Jacobs, "Automatic Thumbnail Cropping and Its Effectiveness," in 16th Annual ACM Symposium on User Interface Software and Technology. 2003, pp What is the smallest crop that exceeds a specified saliency threshold (proportion of the total image saliency)? – Can be calculated by brute force, but inefficient

A greedy approach User specifies a target output size and saliency threshold. Select the most salient region. Until we reach our threshold: – Expand selection to include next most salient region. – Adjust selection to match aspect ratio of output size. Not guaranteed optimal crop – But much faster – Guaranteed to include whole regions

Demonstration Bounding box in red, correct aspect ratio in green

Determining Saliency Threshold Automatically Find the point of diminishing returns: when does increasing the crop area not become worth it? One simple method: select the image coming after the biggest increase in saliency.

Results OriginalThumbnail

Results OriginalThumbnail

Results OriginalThumbnail

Results OriginalThumbnail

Results OriginalThumbnail

Results OriginalThumbnail

Results OriginalThumbnail

Results OriginalThumbnail

Results OriginalThumbnail

Further possible improvements Usability testing – Test users on how quickly they can locate a specified image in a set given a table of thumbnails. Tweak region parameters Saliency threshold selection – Try out more sophisticated metrics Basic feature recognition – Face detection