A Framework for Photo-Quality Assessment and Enhancement based on Visual Aesthetics Subhabrata Bhattacharya Rahul Sukthankar Mubarak Shah

Reference

Outline Introduction Learning Aesthetics Enhancing Composition Experimental Results

Introduction Assessing the quality of photographs is challenging. Experienced photographers adhere to several rules of composition.  Rule of Thirds  Visual Weight Balance

Subject of interest is aligned to one of the stress points. Rule of Thirds

Rule of Thirds : Example

In a well composed image the visual weights of different regions satisfy the Golden Ratio. Visual Weight Balance Sea Sky k ~1.618k

Visual Weight Balance : Example

Introduction In this paper, will use these two rules to assess an image. Formulate photo quality evaluation as a machine learning problem.

Overview

Learning Aesthetics Dataset User Survey Aesthetic Features Learning and Prediction

Dataset Single subject Compositions (384)Landscapes/Seascapes (248)

User Survey 15 participants were asked to assign integer rank from 1 to 5. Each user was asked to rank no more than 30 images. Generate single ground truth for each image (F a ).

User Survey

Aesthetic Feature Extract a relative foreground position feature for images with single-foreground compositions. A visual weight ratio feature for photographs of seascapes or landscapes.

Defined as the normalized Euclidean distance between foreground’s mass to each four stress points. Relative foreground position

The ratio of the sky region, to that in the support region ( ground or sea). Visual weight ratio YgYg YkYk

Learning and Prediction We use SVR to learn the mappings. 150 random images for training and resting for testing.

Enhancing Composition Relocate the foreground object to increase the predicted appeal factor. Better balancing the visual weights of the sky and support region.

Why Cropping does not work? Optimal Crop

Semantic Segmentation Input Image Geometric Context Classifier* *D. Hoiem, A.A. Efros, and M. Hebert, "Geometric Context from a Single Image", ICCV 2005 Sky Support Post Processing Horizon Segmented Foreground

Optimal object placement Support Neighborhood s.t. neighbors stay “like neighbors” + Intensity Term Gradient Term

Example PAF = 3.22 Original Image PAF = 4.53 Optimal Solution

Rescaling Scaling Factor Vanishing Point Optimal location Visual Attention Center

Inpainting Foreground Hole Yunjun Zhang. Jiangjian Xiao. Mubarak Shah, “Region Completion in a Single Image”, EUROGRAPHICS 04 Inpaint Hole

Balancing visual weights YkYk YgYg Ratio of Current extents Y k +h YgYg h = vertical extent of the balanced image

Experimental Results PAF = 3.77 PAF = 4.25 Before After

Experimental Results PAF = 3.92PAF = 4.11 Before After

PAF = 3.98 PAF = 4.46 Experimental Results Before After

PAF = 4.02PAF = 4.34 BeforeAfter Experimental Results

PAF = 3.13 PAF = 4.19 Experimental Results BeforeAfter

PAF = 3.83 PAF = 4.02 Experimental Results Before After

PAF = 3.92 PAF = 4.38 Experimental Results Before After

Experimental Results PAF = 4.02 PAF = 4.71 Before After

Experimental Results PAF = 4.17 PAF = 4.49 Before After

Optimal Placement

Visual Weights

Failure case PAF = 2.34 Fa = 2.41 (Ground Truth) Before PAF = 3.63Fa = 2.54 (Ground Truth) After

Thank You