1. Invariant Large Margin Nearest Neighbour Classifier M. Pawan Kumar Philip Torr Andrew Zisserman

2. Aim To learn a distance metric for invariant nearest neighbour classification Training data

3. Aim To learn a distance metric for invariant nearest neighbour classification Target pairs

4. Aim To learn a distance metric for invariant nearest neighbour classification Impostor pairs Problem : Euclidean distance may not provide correct nearest neighbours Solution : Learn a mapping to new space

5. Aim To learn a distance metric for invariant nearest neighbour classification Bring Target pairs closer Move Impostor pairs away

6. Aim Euclidean DistanceLearnt Distance To learn a distance metric for invariant nearest neighbour classification

7. Aim Euclidean DistanceLearnt Distance To learn a distance metric for invariant nearest neighbour classification

8. Aim Euclidean DistanceLearnt Distance To learn a distance metric for invariant nearest neighbour classification

9. Aim Euclidean DistanceLearnt Distance To learn a distance metric for invariant nearest neighbour classification

10. Aim Euclidean DistanceLearnt Distance To learn a distance metric for invariant nearest neighbour classification

11. Aim To learn a distance metric for invariant nearest neighbour classification Transformation Trajectories Learn a mapping to new space

12. Aim To learn a distance metric for invariant nearest neighbour classification Bring Target Trajectory pairs closer Move Impostor Trajectory pairs away

13. Aim Euclidean DistanceLearnt Distance To learn a distance metric for invariant nearest neighbour classification

14. Motivation Face Recognition in TV Video I1I1 I2I2 I3I3 I4I4 InIn...... Feature Vector Euclidean distance may not give correct nearest neighbours Learn a distance metric

15. Motivation Face Recognition in TV Video Invariance to changes in position of features

16. Large Margin Nearest Neighbour (LMNN) Preventing Overfitting Polynomial Transformations Invariant LMNN (ILMNN) Experiments Outline

17. LMNN Classifier Weinberger, Blitzer and Saul - NIPS 2005 Learns a distance metric for Nearest Neighbour classification Learns a mapping L x Lx Bring target pairs closer Move impostor pairs away xixi xjxj xkxk

18. LMNN Classifier Weinberger, Blitzer and Saul - NIPS 2005 Learns a distance metric for Nearest Neighbour classification Distance between x i and x j : D(i,j) = (x i -x j ) T L T L (x i -x j ) xixi xjxj xkxk

19. LMNN Classifier Weinberger, Blitzer and Saul - NIPS 2005 Learns a distance metric for Nearest Neighbour classification Distance between x i and x j : D(i,j) = (x i -x j ) T M (x i -x j ) min Σ ij D(i,j) subject to M 0 Convex Semidefinite Program (SDP) M 0 xixi xjxj xkxk Global minimum

20. LMNN Classifier Weinberger, Blitzer and Saul - NIPS 2005 Learns a distance metric for Nearest Neighbour classification D(i,k) – D(i,j) ≥ 1- e ijk e ijk ≥ 0 min Σ ijk e ijk subject to M 0 Convex SDP xixi xjxj xkxk

21. LMNN Classifier Weinberger, Blitzer and Saul - NIPS 2005 Learns a distance metric for Nearest Neighbour classification min Σ ij D(i,j) + Λ H Σ ijk e ijk subject to M 0 D(i,k) – D(i,j) ≥ 1- e ijk e ijk ≥ 0 Solve to obtain optimum M Complexity : Polynomial in number of points

22. LMNN Classifier Weinberger, Blitzer and Saul - NIPS 2005 Advantages Trivial extension to multiple classes Efficient polynomial time solution Disadvantages Large number of degrees of freedom – overfitting ?? Does not model invariance of data

23. Large Margin Nearest Neighbour (LMNN) Preventing Overfitting Polynomial Transformations Invariant LMNN (ILMNN) Experiments Outline

24. L 2 Regularized LMNN Classifier Regularize Frobenius norm of L ||L|| 2 = Σ M ii min Σ ij D(i,j) + Λ H Σ ijk e ijk + Λ R Σ i M ii subject to M 0 D(i,k) – D(i,j) ≥ 1- e ijk e ijk ≥ 0 L 2 -LMNN

25. Diagonal LMNN Learn a diagonal L matrix => Learn a diagonal M matrix min Σ ij D(i,j) + Λ H Σ ijk e ijk subject to M 0 D(i,k) – D(i,j) ≥ 1- e ijk e ijk ≥ 0 M ij = 0, i ≠ j Linear Program D-LMNN

26. Diagonally Dominant LMNN Minimize 1-norm of off-diagonal element of M min Σ ij D(i,j) + Λ H Σ ijk e ijk + Λ R Σ ij t ij subject to M 0 D(i,k) – D(i,j) ≥ 1- e ijk e ijk ≥ 0 t ij ≥ M ij, t ij ≥ -M ij, i ≠ j DD-LMNN

27. LMNN Classifier What about invariance to known transformations? Append input data with transformed versions InefficientInaccurate Can we add invariance to LMNN? No – Not for a general transformation Yes - For some types of transformations

28. Large Margin Nearest Neighbour (LMNN) Preventing Overfitting Polynomial Transformations Invariant LMNN (ILMNN) Experiments Outline

29. Polynomial Transformations x = a b Rotate x by an angle θ a b cos θ sin θ -sin θ cos θ 1-θ 2 /2 -(θ-θ 3 /6)a b (θ-θ 3 /6) 1-θ 2 /2 Taylor’s Series

30. Polynomial Transformations x = a b Rotate x by an angle θ a b cos θ sin θ -sin θ cos θ a 1 θ b-a/2b/6 ba-b/2-a/6 θ2θ2 θ3θ3 Xθ T(θ,x) = X θ

31. Why are Polynomials Special? ≡ P 0 θ1θ1 θ2θ2 (θ 1,θ 2 ) DISTANCEDISTANCE Sum of squares of polynomials SD-Representability of PolynomialsLasserre, 2001

32. Why are Polynomials Special? ≡ P’ 0 θ1θ1 θ2θ2 DISTANCEDISTANCE Sum of squares of polynomials

33. Large Margin Nearest Neighbour (LMNN) Preventing Overfitting Polynomial Transformations Invariant LMNN (ILMNN) Experiments Outline

34. ILMNN Classifier Learns a distance metric for invariant Nearest Neighbour classification Learns a mapping L x Lx Bring target trajectories closer Move impostor trajectories away Polynomial trajectories xixi xjxj xkxk

35. ILMNN Classifier Learns a distance metric for invariant Nearest Neighbour classification Learns a mapping L x Lx Bring target trajectories closer Move impostor trajectories away Polynomial trajectories M 0 Minimize maximum distance Maximize minimum distance xixi xjxj xkxk

36. ILMNN Classifier Learns a distance metric for invariant Nearest Neighbour classification Use SD-Representability. One Semidefinite Constraint. Polynomial trajectories Solve for M in polynomial time. Add regularizers to prevent overfitting. xixi xjxj xkxk

37. Large Margin Nearest Neighbour (LMNN) Preventing Overfitting Polynomial Transformations Invariant LMNN (ILMNN) Experiments Outline

38. Dataset Faces from an episode of “Buffy – The Vampire Slayer” 11 Characters * Thanks to Josef Sivic and Mark Everingham 24,244 Faces (with ground truth labelling*)

39. Dataset Splits Experiment 1 Experiment 2 Random permutation of dataset 30% training 30% validation (to estimate Λ H and Λ R ) 40% testing First 30% training Next 30% validation Last 40% testing Suitable for Nearest Neighbour-type Classification Not so suitable for Nearest Neighbour-type Classification

40. Incorporating Invariance Invariance of feature position to Euclidean Transformation -5 o ≤ θ ≤ 5 o -3 ≤ t x ≤ 3 pixels -3 ≤ t y ≤ 3 pixels Approximated to degree 2 polynomial using Taylor’s series Derivatives approximated as image differences Image Rotated Image

41. Incorporating Invariance Invariance of feature position to Euclidean Transformation -5 o ≤ θ ≤ 5 o -3 ≤ t x ≤ 3 pixels -3 ≤ t y ≤ 3 pixels Approximated to degree 2 polynomial using Taylor’s series Derivatives approximated as image differences Smooth Image - = Derivative

42. Training the Classifiers Within-shot Faces Problem : Euclidean distance provides 0 error Solution : Cluster.

43. Training the Classifiers Efficiently solve SDP using Alternative Projection Bauschke and Borwein, 1996 Problem : Euclidean distance provides 0 error Solution : Cluster. Train using cluster centres.

44. Testing the Classifiers Map all training points using L Map the test point using L Find nearest neighbours. Classify. Measure Accuracy = No. of True Positives No. of Test Faces

45. Timings MethodTrainingTesting kNN-E-62.2 s L 2 -LMNN4 h62.2 s D-LMNN1 h53.2 s DD-LMNN2 h50.5 s L 2 -ILMNN24 h62.2 s D-ILMNN8 h48.2 s DD-ILMNN24 h51.9 s M-SVM300 s446.6 s SVM-KNN -2114.2 s

46. Accuracy MethodExperiment 1Experiment 2 kNN-E83.626.7 L 2 -LMNN61.222.6 D-LMNN85.624.3 DD-LMNN84.424.5 L 2 -ILMNN65.924.0 D-ILMNN87.232.0 DD-ILMNN86.629.8 M-SVM62.330.0 SVM-KNN 75.528.1

47. Accuracy MethodExperiment 1Experiment 2 kNN-E83.626.7 L 2 -LMNN61.222.6 D-LMNN85.624.3 DD-LMNN84.424.5 L 2 -ILMNN65.924.0 D-ILMNN87.232.0 DD-ILMNN86.629.8 M-SVM62.330.0 SVM-KNN 75.528.1

48. Accuracy MethodExperiment 1Experiment 2 kNN-E83.626.7 L 2 -LMNN61.222.6 D-LMNN85.624.3 DD-LMNN84.424.5 L 2 -ILMNN65.924.0 D-ILMNN87.232.0 DD-ILMNN86.629.8 M-SVM62.330.0 SVM-KNN 75.528.1

49. Accuracy MethodExperiment 1Experiment 2 kNN-E83.626.7 L 2 -LMNN61.222.6 D-LMNN85.624.3 DD-LMNN84.424.5 L 2 -ILMNN65.924.0 D-ILMNN87.232.0 DD-ILMNN86.629.8 M-SVM62.330.0 SVM-KNN 75.528.1

50. True Positives

51. Conclusions Regularizers for LMNN Adding invariance to LMNN More accurate than Nearest Neighbour More accurate than LMNN

52. Future Research D-LMNN and D-ILMNN for Chi-squared distance D-LMNN and D-ILMNN for dot product distance Handling missing data – Sivaswamy, Bhattacharya, Smola, JMLR – 2006 Learning local mappings (adaptive kNN)

53. Questions ??

55. False Positives

56. Precision-Recall Curves Experiment 1

57. Precision-Recall Curves Experiment 1

58. Precision-Recall Curves Experiment 1

59. Precision-Recall Curves Experiment 2

60. Precision-Recall Curves Experiment 2

61. Precision-Recall Curves Experiment 2