Convex Quadratic Programming for Object Location Hao Jiang, Mark S. Drew and Ze-Nian Li School of Computing Science Simon Fraser University

2 Introduction  Object localization is an important task in computer vision Template

3 Object localization and labeling  Object localization can be formulated as labeling problems.  Consistent labeling  Find small cost label assignment.  Enforce labeling consistency of neighboring sites. p q fpfp fqfq c(p,f p ) c(q,f q ) TemplateTarget Object Site Label Neighboring relation (q-p) (f q -f p )

4 Previous methods for consistent labeling  Consistent labeling, in general form, is NP-hard.  Polynomial time schemes exist for special cases  Dynamic Programming (DP).  Max-flow [Ishikawa 2000, Roy 98].  Approximation schemes  Greedy (local searching) methods  Relaxation labeling (RL) [Rosenfeld 76].  Iterated Conditional Modes (ICM) [Besag 86]. Need good initialization and easily trapped in local minimum.  Global searching methods  Graduated Non-Convexity (GNC) [Blake & Zisserman 87]  Belief Propagation (BP) [Pearl 88, Weiss 2001].  Graph Cut (GC) [Boykov & Zabih 2001].

5 Focus of the research (# of labels in the order of hundred) (# of labels can be in the order of thousand or bigger) Many vision problems such as object matching, large scale motion, tracking etc will benefit from the solver. Well solved Hard to solve The focus of the research Consistent labeling  Previous methods become slow as the number of labels goes to the order of several thousand.

6 The trick of the proposed scheme  In stead of working on the original label space, we represent the label set with a small number basis labels.  We convert the hard problem into a sequence of simpler problems built using only the basis labels.  In this way, the size of the approximation problem is largely decoupled from the original label set.  Each sub-problem is a convex problem and can be globally optimized.  A successive relaxation implementation is used to zero in the target.

7 The non-linear optimization problem  The labeling problem can be solved by optimizing: p q fpfp fqfq c(p,f p ) c(q,f q ) (q-p) (f q -f p )

8 Convex relaxation  To convert the labeling cost term into linear functions, for each site, we define a basis label set.  Each label can then be represented as a linear combination of the basis labels.  The cost of the label is approximated by the linear combination of the costs of these basis labels. t c(s,t) J1J1 J2J2 fsfs f s = a * J 1 + b * J 2 c(s,f s ) = a * c(s,J 1 ) + b * c(s,J 2 ) a + b = 1

9 Convex relaxation (Cont’)  The L2 norm smoothness terms do not need additional conversions. j2j2 j3j3 j4j4 j5j5 j1j1 j 1  s,j 1 + j 2  s,j 2 + j 3  s,j 3 + j 4  s,j 4 + j 5  s,j 5 c(s,j 1 ) c(s,j 2 ) c(s,j 3 ) c(s,j 4 ) c(s,j 5 ) c(s,j 1 )  s,j 1 + c(s,j 2 )  s,j 2 + c(s,j 3 )  s,j 3 + c(s,j 4 )  s,j 4 + c(s,j 5 )  s,j 5  s,j 1 +  s,j 2 +  s,j 3 +  s,j 4 +  s,j 5 =1

10 Convex quadratic program (CQP) We have the convex quadratic program:

11 In some cases, the CQP is exact  If  are binary numbers, the mixed integer program is exactly equivalent to the original problem.  If c(s,t) is convex over t for each s 2 S, the CQP is equivalent to the continuous extension of the original problem. t t c(s,t) Feasible solution Continuous extension

12 In general it is an approximation  For general problems, the CQP approximates each labeling cost surface with the 3D lower convex hull. t =(x,y) c(s,t) The convexified surfaces are much simpler than the original ones. x y Label cost

13 The CQP can be greatly simplified  The most compact basis labels correspond to the lower convex hull vertices.  This shows a way to simplify the label set in labeling.  We can safely discard many labels without worrying about the problems met in previous methods. # of original label: 400 # of basis label: 20

14 Successive convexification To improve the approximation we use the successive convexification (SC) scheme as follows:

15 Experimental results Matching Random Dots

16 Experimental results (Cont’) Matching Random Dots

17 Experimental results (Cont’) Matching Leaf

18 Experimental results (Cont’) Matching Face

19 Experimental results (Cont’) Matching Hand

20 Conclusion  Set out an object localization method which can deal with textureless objects in strong background clutter.  Propose a successive convex quadratic method  CQP large decouples the number of target image points with the size of the convex program – It searches the whole target image quickly.  Successive convexification is studied to improve the result recursively.  Successive convexification is a general method can be applied to any convex regularization problem.

21 Thank you!

22 In summery  Label space is approximated with small number of basis labels.  Original problem is converted to a sequence of much easier convex programs and solved by successive relaxation.  The size of the convex program is largely decoupled with the number of candidate labels Convexification Trust Region Shrinking Template Target Image Matching Costs Lower Convex Hull