The University of Ontario CS 4487/9687 Algorithms for Image Analysis Multi-Label Image Analysis Problems

The University of Ontario CS 4487/9687 Algorithms for Image Analysis Multi-label image analysis problems n Topic 1 From binary to multi-label problems: Stereo, image restoration, texture synthesis, multi-object segmentation Ishikawa’s algorithm, total variation n Topic 2 Types of pair-wise pixel interactions Convex interactions Discontinuity preserving interactions n Topic 3 Energy minimization algorithms: simulated annealing, ICM, a-expansions Extra Reading: …

The University of Ontario Graph cuts algorithms can minimize multi-label energies as well

The University of Ontario Multi-scan-line stereo with s-t graph cuts (Roy&Cox’98) x y

The University of Ontario Multi-scan-line stereo with s-t graph cuts (Roy&Cox’98) s t cut L(p) p “cut” x y labels x y Disparity labels

The University of Ontario s-t graph-cuts for multi-label energy minimization n Ishikawa 1998, 2000, 2003 n Modification of construction by Roy&Cox 1998 V(dL) dL=Lp-Lq V(dL) dL=Lp-Lq Linear interactions “Convex” interactions

The University of Ontario Pixel interactions V: “convex” vs. “discontinuity-preserving” V(dL) dL=Lp-Lq Potts model Robust “discontinuity preserving” Interactions V V(dL) dL=Lp-Lq “Convex” Interactions V V(dL) dL=Lp-Lq V(dL) dL=Lp-Lq “linear” model

The University of Ontario Pixel interactions: “convex” vs. “discontinuity-preserving” “linear” V truncated “linear” V

The University of Ontario code Robust interactions n NP-hard problem (3 or more labels) two labels can be solved via s-t cuts n a-expansion approximation algorithm (Boykov, Veksler, Zabih 1998, 2001) guaranteed approximation quality (Veksler, 2001) –within a factor of 2 from the global minima (Potts model) n Many other (small or large) move making algorithms - a/b swap, jump moves, range moves, fusion moves, etc. n LP relaxations, message passing, e.g. (LBP, TRWS) n Other MRF techniques (simulated annealing, ICM) n Variational methods (e.g. multi-phase level-sets)

The University of Ontario other labels a a-expansion move Basic idea is motivated by methods for multi-way cut problem (similar to Potts model) Break computation into a sequence of binary s-t cuts

The University of Ontario a-expansion (binary move) optimizies sumbodular set function expansions correspond to subsets (shaded area) L current labeling =

The University of Ontario a-expansion (binary move) optimizies sumbodular set function L current labeling 0 1

The University of Ontario a-expansion (binary move) optimizies sumbodular set function L current labeling

The University of Ontario a-expansion (binary move) optimizies sumbodular set function L current labeling Set function is submodular if

The University of Ontario a-expansion (binary move) optimizies sumbodular set function L current labeling Set function is submodular if = 0 triangular inequality for ||a-b||=E(a,b)

The University of Ontario a-expansion (binary move) optimizies sumbodular set function L current labeling a-expansion moves are submodular if is a metric on the space of labels [Boykov, Veksler, Zabih, PAMI 2001]

The University of Ontario a-expansion algorithm 1.Start with any initial solution 2.For each label “a” in any (e.g. random) order 1.Compute optimal a-expansion move (s-t graph cuts) 2.Decline the move if there is no energy decrease 3.Stop when no expansion move would decrease energy

The University of Ontario a-expansion moves initial solution -expansion In each a-expansion a given label “a” grabs space from other labels For each move we choose expansion that gives the largest decrease in the energy: binary optimization problem

The University of Ontario Multi-way graph cuts stereo vision original pair of “stereo” images depth map ground truth BVZ 1998 KZ 2002

The University of Ontario normalized correlation, start for annealing, 24.7% err simulated annealing, 19 hours, 20.3% err a-expansions (BVZ 89,01) 90 seconds, 5.8% err a-expansions vs. simulated annealing

The University of Ontario a-expansions: examples of metric interactions Potts V “noisy diamond” “noisy shaded diamond” Truncated “linear” V

The University of Ontario Multi-way graph cuts Graph-cut textures (Kwatra, Schodl, Essa, Bobick 2003) similar to “image-quilting” (Efros & Freeman, 2001) A B C D E F G H I J A B G D C F H I J E

The University of Ontario Multi-way graph cuts Graph-cut textures (Kwatra, Schodl, Essa, Bobick 2003)

The University of Ontario Multi-way graph cuts Multi-object Extraction Obvious generalization of binary object extraction technique (Boykov, Jolly, Funkalea 2004)

The University of Ontario Block-coordinate descent alternating a-expansion (for segmentation L ) and fitting colors I i Chan-Vese segmentation (multi-label case) Potts model

The University of Ontario Chan-Vese segmentation (multi-label case) Block-coordinate descent alternating a-expansion (for segmentation L ) and fitting colors I i Potts model

The University of Ontario Block-coordinate descent alternating a-expansion (for segmentation L ) and fitting colors I i Stereo via piece-wise constant plane fitting [Birchfield &Tomasi 1999] Models T = parameters of affine transformations T(p)=a p + b 2x2 2x1 Potts model

The University of Ontario Block-coordinate descent alternating a-expansion (for segmentation L ) and fitting colors I i Piece-wise smooth local plane fitting [Olsson et al. 2013] truncated angle-differences non-metric interactions need other optimization

The University of Ontario Block-coordinate descent alternating a-expansion (for segmentation L ) and fitting colors I i Signboard segmentation [Milevsky 2013] Labels = planes in RGBXY space C(p) = a x + b Potts model 3x2 3x1

The University of Ontario Signboard segmentation [Milevsky 2013] 3x2 3x1 Goal: detection of characters, then text line fitting and translation

The University of Ontario Multi-label optimization n 80% of computer vision and bio-medical image analysis are ill-posed labeling problems requiring optimization of regularization energies E(L) n Most problems are NP hard n Optimization algorithms is area of active research Google, Microsoft, GE, Siemens, Adobe, etc. LP relaxations [Schlezinger, Komodakis, Kolmogorov, Savchinsky,…] Message passing, e.g. LBP, TRWS [Kolmogorov] Graph Cuts (a-expanson, a/b-swap, fusion, FTR, etc) Variational methods