1. Robust Real Time Pattern Matching using Bayesian Sequential Hypothesis Testing Ofir PeleMichael Werman The Hebrew University of Jerusalem TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A AA A A A AA A AAA A

2. Goal of Work Find patterns in images In real time Under different transformations: – Light changes – Occlusion – Non rigid deformations – (small) geometrical transforms

8. Pattern Occurrence of pattern in image

9. Pattern Occurrence of pattern in image

10. Performance Image Size: 640x480 pixels 0.037sec Online Time: 0.037 sec

11. Related Work – Fast Pattern Matching Scale Invariant Features – Lowe 04 Normalized Cross Correlation – Lewis 95 Fast filters for norm distances – Cha 00, Hel-Or and Hel-Or 05 Sequential sampling – Barnea and Silverman 72, Pereira and Mascarenhas 84 – Our method

12. Contributions A simple and robust family of distance measures, the “Image Hamming Distance Family” + Bayesian framework for sequential hypothesis testing on finite populations. = Robust Real Time Pattern Matching

13. Sliding Window Approach

14. Sliding Window Approach Questions Which similarity measure ? Can we apply it fast ? Can we use information from one window to the next ?

15. Sliding Window Approach Questions Which similarity measure ? Can we apply it fast ? Can we use information from one window to the next ?

16. Sliding Window Approach Questions Which similarity measure - Hamming Can we apply it fast ? Can we use information from one window to the next ?

17. Hamming Distance on Pixels

18. Hamming Distance on Pairs of Pixels

19. Advantages of Hamming Distance Pluggable invariance: – Noise that preserve monotonic relations – Small non-rigid deformations – … Inherent robustness to noise that change the value of pixels completely: – Occlusions – Out of plane rotations – Shading – Spectral reflactance

20. Sliding Window Approach Questions Which similarity measure ? Can we apply it fast ? Can we use information from one window to the next ?

21. Sliding Window Approach Questions Which similarity measure ? Can we apply it fast - Sample Can we use information from one window to the next ?

22. Related Work – SSDA (Barnea and Silverman 72)

25. Return non-similar Pairs of Pixels Examined Cumulative Difference Related Work – SSDA (Barnea and Silverman 72)

26. Return non-similar Pairs of Pixels Examined Cumulative Difference Pairs of Pixels Examined Related Work – SSDA (Barnea and Silverman 72)

27. Which distance measure to use ? How to design the sampling algorithm ? Related Work – SSDA (Barnea and Silverman 72)

28. Related Work – Wald’s SPRT (Pereira and Mascarenhas 84) Sample while:

29. Related Work – Wald’s SPRT (Pereira and Mascarenhas 84) Wald’s approximation:

30. 2. Assume everything Gaussian  Related Work – Wald’s SPRT (Pereira and Mascarenhas 84) How to model ? 1. Transform to binary images 

31. Designing an Optimal Sampling Algorithm

32. Reduction To Hypothesis Testing Template: Sub Image: 0 0 0 1

33. Reduction To Hypothesis Testing Template: Image: … ??? ???

34. The Decision Matrix 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar Pairs of Pixels Examined Cumulative difference

35. Rules of Game Each coin check cost time units Each check of sack with coins costs time units ???????????????

36. Optimization

37. Computing Probabilities 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar n = Pairs of Pixels Examined k = Cumulative difference ??

38. Non-similar pairs of pixels sampled Similar pixels sampled Computing Probabilities

39. 1 3 3 1 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar Computation of 11 12 10 0 1 0 n = Pairs of Pixels Examined k = Cumulative difference

40. Computing Probabilities We define:

41. Computing Probabilities – Closure

43. Optimization

44. Auxiliary problem

45. ? ? ? ? ? ? ? ? ? ? ? 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar n = Pairs of Pixels Examined k = Cumulative difference Solving Auxiliary problem - Backward Induction Stop and return non-similar Stop and return similar Do the exact computation

46. 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar n = Pairs of Pixels Examined k = Cumulative difference Solving Auxiliary problem - Backward Induction

47. ? ? ? ? ? ? ? ? ? ? ? 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar n = Pairs of Pixels Examined k = Cumulative difference Solving Auxiliary problem - Backward Induction Stop and return non-similar Stop and return similar Do the exact computation Continue sampling

48. 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar n = Pairs of Pixels Examined k = Cumulative difference Solving Auxiliary problem - Backward Induction

49. 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar n = Pairs of Pixels Examined k = Cumulative difference Solving Auxiliary problem - Backward Induction

50. 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar n = Pairs of Pixels Examined k = Cumulative difference Solving Auxiliary problem - Backward Induction

51. Solving Optimization using Auxiliary problem Claim:

52. Solving Optimization using Auxiliary problem Claim:

53. Solving Optimization using Auxiliary problem Proof:

54. Solving Optimization using Auxiliary problem

58. Claim:

59. Solving Optimization using Auxiliary problem

60. Finding the Right Error Weights

61. Offline Time Complexity

62. Practical Observation 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar Pairs of Pixels Examined Cumulative difference The events of returning similar or doing the exact computation are very rare. There is a cost to check them.

63. Practical Observation 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar Pairs of Pixels Examined Cumulative difference This matrix has a higher expected number of samples, but a smaller expected running time because of the decrease in the cost of each sample,

64. Can we Reduce Offline Time ?

65. SPRT for Composite Hypotheses Sample both images as long as:

66. SPRT for Composite Hypotheses Wald’s approximation:

67. SPRT for Composite Hypotheses Computation of likelihood ratio: Numerical stability

68. SPRT for Composite Hypotheses Assuming correct prior, our hypotheses are actually simple – there are no free parameters The proof for sub-optimality of SPRT is valid Sub-optimal because of “overshoot” effect Excellent practical results

69. Designing the SPRT 0123456789101112131415161718192021 1 2 3 4 5 6 7 8 9 10 0 Continue sampling Stop and return similarDo the exact computation Stop and return non-similar Pairs of Pixels Examined Cumulative difference ? ?

70. SPRT Offline Time Complexity is updated each stage. time complexity and memory complexity Practical time of 0.067 seconds for

71. Estimating the Prior 480 Training images Random, not too smooth, pattern

72. Estimating the Prior

73. Experiments That Checks Sampling Performance 480 Test Images Random, not too smooth, pattern

74. Fixed Size Sampling vs Sequential Sampling 30x30 pattern size Thresholded Absolute Difference Distance Estimated prior

75. Fixed Size Sampling vs Sequential Sampling 30x30 pattern size Thresholded Absolute Difference Distance Uniform prior

76. Estimated Prior vs Uniform Prior 60x60 pattern size Threshold l 2 norm in L*a*b color space OPT

77. SPRT vs OPT 30x30 pattern size Thresholded Absolute Difference Distance Estimated prior

78. Gaussian Noise Image copyright by Ben Schumin. Taken from: http://en.wikipedia.org/wiki/Image:July_4_crowd_at_Vienna_Metro_station.jpg Gaussian noise with mean 0 and variance of 0.01

79. Gaussian Noise Image copyright by Ben Schumin. Taken from: http://en.wikipedia.org/wiki/Image:July_4_crowd_at_Vienna_Metro_station.jpg

80. Gaussian Noise Image copyright by Ben Schumin. Taken from: http://en.wikipedia.org/wiki/Image:July_4_crowd_at_Vienna_Metro_station.jpg

81. Gaussian Noise Pattern Occurrences of pattern in image Image copyright by Ben Schumin. Taken from: http://en.wikipedia.org/wiki/Image:July_4_crowd_at_Vienna_Metro_station.jpg

82. Distance Used

83. Performance Image Size: 640x480 pixels Pattern Size: 1089 pixels Average Sample Size: 17.86 pixels Offline Time: 0.018 sec 0.019sec Online Time: 0.019 sec

84. Light Changes + Rotation

86. Pattern Occurrences of pattern in image

87. Light Changes + Rotation Pattern Occurrences of pattern in image

88. Distance Used Monotonic Relations Neighbors pixels with absolute intensity value difference greater than 80

89. Performance Image Size: 640x480 pixels Pattern Size: 631 pairs of pixels Average Sample Size: 35.28 pairs of pixels Offline Time: 0.009 sec 0.021sec Online Time: 0.021 sec

90. Occlusion + Rotation

92. Pattern Occurrences of pattern in images

93. Occlusion + Rotation Pattern Occurrences of pattern in images

94. Distance Used

95. Performance Image Size: 640x480 pixels Pattern Size: 2197 pixels Average Sample Size: 19.7 pixels Offline Time: 0.067 sec 0.022sec Online Time: 0.022 sec No motion considerations.

96. Non Rigid Deformations

99. Pattern Occurrence of pattern in image

100. Non Rigid Deformations Pattern Occurrence of pattern in image

101. Distance Used

102. Performance Image Size: 640x480 pixels Pattern Size: 714 pixels Average Sample Size: 17.23 pixels Offline Time: 0.007 sec 0.064sec Online Time: 0.064 sec

103. Non Rigid Deformations – comparison to SIFT

105. Performance Image Size: 640x480 pixels Pattern Size: 9409 pairs of pixels Average Sample Size: 39.98 pairs of pixels Offline Time: 1.219 sec 0.037sec Online Time: 0.037 sec

106. Sliding Window Approach Questions Which similarity measure ? Can we apply it fast ? Can we use information from one window to the next ?

107. Sliding Window Approach Questions Which similarity measure ? Can we apply it fast ? Can we use information from one window to the next – Using smoothness

108. LU Rank of Pixels

109. Computing LU Rank of Pixels

110. Acknowledgments Professor Ester Samuel-Cahn Refael Vivanti Amichai Zisken, Ori Maoz, Amit Gruber, Amnon Aaronsohn, Aviv Hurvitz and Eran Maryuma

111. Hamming Distance + Sequential Sampling = Robust Real Time Pattern Matching