Models for Scene Understanding – Global Energy models and a Style-Parameterized boosting algorithm (StyP-Boost) Jonathan Warrell, 1 Simon Prince, 2 Philip Torr, 1 Lubor Ladicky, 1 Chris Russell 1 1 Oxford Brookes University, 2 University College London

Overview CRF-based semantic segmentation Recent models Detectors Stereo Co-occurence Hierarchical Energies Style parameterized boosting (StyP-Boost) Open questions / problems

CRF-based semantic segmentation Semantic segmentation = dense labeling using fixed object set

CRF-based semantic segmentation Conditional Random Field model (pairwise) Observed Variables Hidden Variables UnaryPairwise

Example:  -expansion Sky House Tree Ground Initialize with Tree Status: Expand GroundExpand HouseExpand Sky Courtesy: Pushmeet Kohli

Move Making Algorithms Search Neighbourhood Current Solution Optimal Move Solution Space Energy

Higher order CRF models Higher order models UnaryPairwiseHigher-order

Segment-based Potentials No. of pixels not taking l in c

Detector-based Potentials Strength of detector response Lubor Ladicky, Paul Sturgess, Karteek Alahari, Chris Russell, Philip H.S. Torr, What,Where & How Many? Combining Object Detectors and CRFs, ECCV 2010 What,Where & How Many? Combining Object Detectors and CRFs

Co-occurrence Potentials Lubor Ladicky, Chris Russell, Pushmeet Kohli, Philip H.S. Torr, Graph Cut based Inference with Co-occurrence Statistics, ECCV, 2010Graph Cut based Inference with Co-occurrence Statistics Global image label set

Joint Stereo + Segmentation Joint potentials Lubor Ladicky, Paul Sturgess, Chris Russell, Sunando Sengupta, Philip H.S. Torr, Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction, BMVC 2010Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction Object only potentials

Hierarchical Energies Lubor Ladicky, Chris Russell, Pushmeet Kohli, Philip H.S. Torr, Associative Hierarchical CRFs for Object Class Image Segmentation, ICCV, 2009.Associative Hierarchical CRFs for Object Class Image Segmentation Energy between levels 1 and 0

Style-based Potentials Jonathan Warrell, Simon Prince, Philip H.S. Torr, StyP-Boost: A Bilinear Boosting Algorithm for Learning Style-Parameterized Classifiers, BMVC, 2010StyP-Boost: A Bilinear Boosting Algorithm for Learning Style-Parameterized Classifiers Style-based unary potential Style 1:Style 2:

TextonBoost (Shotton et al ’09) Image first convolved with 17-d filter bank Vectors are clustered, and assigned to ~150 texton indices

TextonBoost (Shotton et al ’09) Texture-layout features derived from textons Boosted classifier predicts semantic class

DenseBoost (Ladicky et al ’09) DenseBoost extends TextonBoost to include HOG ColourHOG Structure / Motion features State of the art performance on MSRC (Ladicky et al ’09) CamVid (Sturgess et al ’09) Paul Sturgess, Karteek Alahari, Lubor Ladicky, Philip H.S. Torr, Combining Appearance and Structure from Motion Features for Road Scene Understanding, BMVC, 2009Combining Appearance and Structure from Motion Features for Road Scene Understanding Lubor Ladicky, Chris Russell, Pushmeet Kohli, Philip H.S. Torr, Associative Hierarchical CRFs for Object Class Image Segmentation, ICCV, 2009.Associative Hierarchical CRFs for Object Class Image Segmentation

StyP-Boost Framework (Training) Training Set Objective Classifier form Local featuresStyle ParametersTarget vectors

StyP-Boost Framework (Training) Training Set Objective Classifier form Loss for class k Strong learner for class k

StyP-Boost Framework (Training) Training Set Objective Classifier form Weak learner mStyle s

Corel: Styles through clustering Styles found in Corel through clustering 2-styles (98%) 3-styles(96%) 4-styles (89%)

Corel: Styles through clustering Cluster images based on label histograms during training (2-4 clusters) Train classifier to predict cluster from image Use smoothed classifier posteriors as style parameters (training and testing) cluster label

Corel: Qualitative results StyP-Boost reduces noise from classes which don’t co-occur

Corel: Qualitative results StyP-Boost provides better discrimination of co-occuring classes

Corel: Quantitative results Training set Test set

Open questions / Problems Learning from sparsely labeled data Lamp-post Sign

Open questions / Problems Incorporating 3D and Video Image CRF Ground-plan CRF Volumetric CRF

Open questions / Problems Using temporal information Extend detector potentials to include tracking Use global scene variables for times of day, seasons etc.

Further Questions Further Questions?