1. Contextual models for object detection using boosted random fields by Antonio Torralba, Kevin P. Murphy and William T. Freeman

2. Quick Introduction What is this? Now can you tell?

3. Belief Propagation (BP) Network (Pairwise Markov Random Fields)  observed nodes ( y i )

4. Belief Propagation (BP) Network (Pairwise Markov Random Fields)  observed nodes ( y i )  hidden nodes ( x i )

5. Belief Propagation (BP) Network (Pairwise Markov Random Fields)  observed nodes ( y i )  hidden nodes ( x i ) Statistical dependency, called local evidence: Shord-hand

6. Belief Propagation (BP) Statistical dependency: Local evidence Shord-hand Statistical dependency: Compatibility function

7. Belief Propagation (BP) Joint probability

8. Belief Propagation (BP) Joint probability x x1x1 x2x2 xixi …. x5x5 x3x3 x1x1 x4x4 xjxj x 12 y1y1 y2y2 yiyi

9. Belief Propagation (BP) Joint probability x x1x1 x2x2 xixi …. x5x5 x3x3 x1x1 x4x4 xjxj x 12 y1y1 y2y2 yiyi

10. Belief Propagation (BP) The belief b at a node i is represented by  the local evidence of the node  all the messages coming in from neighbors xixi xjxj ∏ NiNi yiyi

11. Belief Propagation (BP) The belief b at a node i is represented by  the local evidence of the node  all the messages coming in from neighbors xixi xjxj ∏ NiNi yiyi

12. Belief Propagation (BP) Messages m between hidden nodes How likely node j thinks it is that node i will be in the corresponding state. xixi xjxj m ji (x i )

13. Belief Propagation (BP) xixi xjxj xkxk xixi xjxj m ji (x i )

14. Conditional Random Field Distribution of the form:

15. Conditional Random Field Distribution of the form:

16. Boosted Random Field Basic Idea:  Use BP to estimate P(x|y)  Use boosting to maximize Log Likelihood of each node wrt to

17. Algorithm: BP Minimize negative log likelihood of training data ( y i ). Label Loss function to minimize:

18. Algorithm: BP Minimize negative log likelihood of training data ( y i ). Label Loss function to minimize:

19. Algorithm: BP Minimize negative log likelihood of training data ( y i ). Label Loss function to minimize:

20. Algorithm: BP xixi xjxj NiNi ∏ yiyi

21. xixi xjxj NiNi ∏ yiyi

22. xixi xjxj NiNi ∏

23. xixi xjxj

24. xixi F : a function of the input data yiyi

25. Algorithm: BP xixi xjxj with yiyi

26. Algorithm: BP xixi xjxj with yiyi

27. Function F xixi yiyi Boosting!  f is the weak learner: weighted decision stumps.

28. Minimization of loss L

30. where

31. Local Evidence: algorithm For t=1..T  Iterate N boost times find the best basis function h update local evidence with update the beliefs update the weights  Iterate N BP times update messages update the beliefs xixi xjxj yiyi

32. Local Evidence: algorithm For t=1..T  Iterate N boost times find the best basis function h update local evidence with update the beliefs update the weights  Iterate N BP times update messages update the beliefs xixi xjxj yiyi

33. Local Evidence: algorithm For t=1..T  Iterate N boost times find the best basis function h update local evidence with update the beliefs update the weights  Iterate N BP times update messages update the beliefs xixi xjxj yiyi

34. Local Evidence: algorithm For t=1..T  Iterate N boost times find the best basis function h update local evidence with update the beliefs update the weights  Iterate N BP times update messages update the beliefs xixi xjxj yiyi

35. Local Evidence: algorithm For t=1..T  Iterate N boost times find the best basis function h update local evidence with update the beliefs update the weights  Iterate N BP times update messages update the beliefs xixi xjxj yiyi

36. Local Evidence: algorithm For t=1..T  Iterate N boost times find the best basis function h update local evidence with update the beliefs update the weights  Iterate N BP times update messages update the beliefs xixi xjxj yiyi

37. Function G By assuming that the graph is densely connected we can make the approximation: Now G is a non-linear additive function of the beliefs:

38. Function G Instead of learningthe function can be learnt with an additive model: weighted regression stumps

39. Function G The weak learner is chosen by minimizing the loss:

40. The Boosted Random Field Algorithm For t=1..T  find the best basis function h for f  find the best basis function for  compute local evidence  compute compatibilities  update the beliefs  update weights xixi xjxj yiyi

41. The Boosted Random Field Algorithm For t=1..T  find the best basis function h for f  find the best basis function for  compute local evidence  compute compatibilities  update the beliefs  update weights xixi b1b1 b2b2 bjbj …

42. Final classifier For t=1..T  update local evidences F  update compatibilities G  compute current beliefs Output classification:

43. Multiclass Detection U: Dictionary of ~2000 images patches V: Same number of image masks

44. Multiclass Detection U: Dictionary of ~2000 images patches V: Same number of image masks At each round t, for each class c for each dictionary entry d there is a weak learner:

45. Function f To take into account different sizes, we first downsample the image and then upsample and OR the scales: which is our function for computing the local evidence.

46. Function g The compatibily function has a similar form:

47. Function g The compatibily function has a similar form: W represent a kernel with all the messages directed to node x, y, c

48. Kernels W Example of incoming messages:

49. Function G The overall incoming messages function is given by:

50. Learning… Labeled dataset of office and street scenes, with each ~100 images  In the first 5 round updated only the local evidence  After the 5th iteration update also the compatibility functions At each round update only F and G of the single object class that reduces the most the multiclass cost.

51. Learning… Biggest objects are detected first because they reduce the error of all classes the fastest:

52. The End

53. Introduction Observed: Picture Dictionary: Dog P(Dog|Pic)

54. Introduction P(Head|Pic i ) P(Tail|Pic i ) P(Front Legs|Pic i ) P(Back Legs|Pic i )

55. Introduction Comp(Head, Legs) Comp(Head, Tail) Comp(F. Legs, B. Legs) Comp(Tail, Legs) Dog!

56. Introduction P(Piraña|Pic i ) Comp(Piraña, Legs)

57. Graphical Models Observation nodes y i Y y i can be a pixel or a patch

58. Graphical Models Hidden Nodes Local Evidence: X Dictionary Shord-hand

59. Graphical Models Compatibility Function: X