MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky

Introduction The goal: model human visual clutter perception. Visual clutter: A “confused collection”, or a “crowded disorderly state”. Increasing visual clutter. Set Size Effect: search performance decreases as set size increases. How to quantify set size?

Introduction What are proto-objects? Regions of locally similar features, they can be objects, object parts, or just pieces that come together to form objects. What is the proto-object clutter model? Segments an image into proto-objects, use the normalized number of proto-objects as the clutter measure of the image.

Stimuli x600 images of real world contexts Selected from the SUN09 Database 6 object-count groups, each contains 15 images Human labeled objects are provided with SUN09 31~40 objects 15 images 51~60 objects 15 images 1~10 objects 15 images 90 images total

Stimuli

Behavioral method Subjects 15 human subjects age from 18 to 32 Method Rank order the 90 images from least to most cluttered Using a Matlab GUI Participants were told to use their own definition of clutter Practice 12 practice images prior to actual testing

Behavioral method 2 displays were used Images were shown at random Bottom monitor subtended a visual angle of 27° x 20° Had the option to correct the ordering A experiment lasted roughly 45~60 min Average pair-wise rater agreement: ρ = 0.692

Computational method 1. superpixel preprocessing K = 600 Need to merge the resulting over-segmentation

Computational method 2. Mean-shift clustering

Computational method 2. Mean-shift clustering, more examples

Computational method 2. Mean-shift clustering in HSV color space Median color of each superpixel

Computational method 3. Merge neighboring superpixels that belong to the same color cluster 600 superpixels 207 proto-objects (0.345)

Computational method Proto-object visualization Fill each proto-object using the median of the member-pixel colors

Results Spearman’s rank order correlation Ρ = 0.814, p < 0.001

Results Robust to different parameter/color space settings Each correlation is computed using the optimal MS bandwidth

Results Comparing to other clutter models

Results More visualized proto-object results

Some further experiments Does visual clutter perception change when viewing images of different sizes? Experiment: Large images vs small images Same 90-image dataset, large images = original 800x600 size; small images = quarter size (200x150) Same behavioral setting 12 practice images Same Matlab GUI

Further experiments Small images subtended a visual angle of 6.75° x 5 ° 20 undergraduate students from SBU Followed the same procedure as the large-image setup Average inter subject correlation: 0.58

Further experiments Highlights Human’s small image rating vs large image rating: ρ = ! Proto-object model’s small image correlation: ρ = 0.852

Further experiments Comparing with other clutter models Proto-object model stayed the most consistent

Conclusion Number of object (set size) is a poor predictor to visual clutter Set size may be better quantified/represented by proto- objects All segmentation-based methods outperformed the feature/non- segmentation based models Can proto-object’s spatial density predict search performance, and/or number estimation? Can proto-object’s spatial distribution predict gaze?