1. Paper Reading Dalong Du Nov.27, 2009

2. Papers Leon Gu and Takeo Kanade. A Generative Shape Regularization Model for Robust Face Alignment. ECCV08. Yan Li, Leon Gu, Takeo Kanade. A Robust Shape Model for Multi-view Car Alignment. CVPR09.

3. A Generative Shape Regularization Model for Robust Face Alignment Leon Gu and Takeo Kanade

4. Outline Author Introduction. Problem Introduction. How to do? Discussion.

5. Outline Author Introduction. Problem Introduction. How to do? Discussion.

6. Author Introduction (1/3) Leon Gu Takeo Kanade( 金出武雄 )

7. Author Introduction (2/3) Leon Gu –Ph.D. candidate in the Computer Science Department at Carnegie Mellon University, advised by Professor Takeo Kanade. –His main research interest is in developing robust and efficient algorithms for object recognition. A common thread has been the focus on reasoning the shape of visual objects from noisy, real-world images, where the uncertainties over image appearance and imaging conditions are prevalent.

8. Author Introduction (3/3) Takeo Kanade –Director of the Robotics Institute of Carnegie Mellon University –Wisdom ：像外行一样思考，像专家一样实践

9. Outline Author Introduction. Problem Introduction. How to do? Discussion.

10. Problem Introduction (1/12) Face Q consists of N landmark points: – –The geometry information of Q decouples into two parts: A canonical shape S –b –e.g. Or other linear or nonlinear methods A transformation –θ –e.g. similarity s, R, t Or Affine or others. b θ

11. Problem Introduction (2/12) Probabilistic Formulation –Generic alignment problem Where –Pose space Θ is free –Shape space is constrained A solution maximizes the posterior A chicken and egg problem –A best solution A max posterior

12. Problem Introduction (3/12) In the Eyes of Computer – On the basis of such “noisy observation”, how can make the best hypothesis (b, θ) ? Reflectance, Occlusion, Image blur, ….. Noisy feature map

13. Deformation Transformation Image Likelihood Problem Introduction (4/12) A Generative hierarchical model –Deformation The magnitude of deformation is controlled by b. The canonical shape S is generated from b through, a process that could be linear or nonlinear. –Transformation The transform could be similarity/affine. –Image Likelihood Varies with the type of image local feature –Profile, local image patch, Haar-like feature… …

14. Deformation Transformation Image Likelihood Problem Introduction (5/12) Baseline Model –Linear Deformation Where –Shape prior, Λ is diagonal. –Isotropic shape noise (Probabilistic PCA) – The average residual variance outside of the subspace, where N is the number of landmark points, M is the subspace dimension. {Φ, μ, σ} are learned from training samples.

15. Deformation Transformation Image Likelihood Problem Introduction (6/12) Baseline Model –Similarity Transform Where –θ={s, R, t} are scale, Rotation, translation coefficients respectively. –Diagonal observation noise » measures the noise level of the observation of n-th landmark point. » »Σ is also learned from training samples.

16. Deformation Transformation Image Likelihood Problem Introduction (7/12) Baseline Model –Observed shape Y is generated from feature point detector. – EM –Q-function: –E-step: compute the statistics that are required to evaluate Q-function. –M-step: maximize Q-function to find the updated shape and pose.

17. Problem Introduction (8/12) Alignment algorithm

18. Problem Introduction (9/12) Problems? –Linear deformation model Cannot handle faces of rare shapes (babies, etc) Cannot handle extreme expressions –Single candidate position for each feature point Best position may be the one with second strongest response –This paper extends the generative framework to handle Large face shape deformation including extreme expressions Multiple candidate positions for each feature point Identify outliers, like occluded feature points.

19. Problem Introduction (10/12) Handling Extreme Expressions

20. Problem Introduction (11/12) Handling Large Occlusion

21. Problem Introduction (12/12) Handling Real World Images

22. Outline Author Introduction. Problem Introduction. How to do? Discussion.

23. How to do? (1/12) Face Q consists of N landmark points: – –The geometry information of Q decouples into two parts: A canonical shape S A similarity transformation –Map S from a common reference frame to the coordinate plane of the image I b θ

24. How to do? (2/12) Make a mixture of constrained Gaussian –Multiple subspace –

25. How to do? (3/12) Allow generate multiple candidate –For n-th landmark K candidate positions denote the whole set of N × K candidates Set a binary N × K matrix h to specify the “true” candidate e.g.

26. How to do? (4/12) A new generative hierarchical model Deformation Transformation Image Likelihood Deformation Transformation Image Likelihood

27. How to do? (5/12) Deformation –Define prior distribution over the shape S as a mixture of Gaussian –Introduce a multinomial distribution z –Model parameters learned from training samples e.g.

28. How to do? (6/12) Similarity Transform Where – θ={s, R, t} are scale, Rotation, translation coefficients respectively. –Diagonal observation noise measures the noise level of the observation of n-th landmark point. Σ is also learned from training samples and can updated on fitting phase. –So

29. How to do? (7/12) Image Likelihood –The image likelihood of seeing a landmark atone particular position Q nk is measured by – is generated by feature detector.

30. How to do? (8/12) Goal: –Solve b and θ on the basis of the candidate point set Q. MAP problem which can be solved by EM – Posterior with latent variables S, h, z –Take the expectation of the log over the posterior of the latent variables S, h, z Q function:

31. How to do? (9/12) Alignment Algorithm

32. How to do? (10/12) Update Canonical Shape

33. How to do? (11/12) Update Shape Parameters Shrink by:

34. How to do? (12/12) Identifying Outliers: –Use observation noise model –Observation noises are unpredictable Update online –Change it according to the fitting error between the model prediction and the averaged candidate position Define weights to update Canonical Shape –A smaller leads a larger weight to the canonical shape and less to the observed candidate.

35. Outline Author Introduction. Problem Introduction. How to do? Discussion.

36. Discussion (1/6) Evaluation

37. Discussion (2/6) Handling Extreme Expressions –Number of mixture components is L = 3.

38. Discussion (3/6) Handling Large Occlusion

39. Discussion (4/6) Handling Real World Images

40. Discussion (5/6) Similarity Transform Where –Diagonal observation noise measures the noise level of the observation of n-th landmark point. The independence assumption to each landmark is not reasonable. –Markov Network –…

41. Discussion (6/6) The regularization step does not consider the image information anymore

42. A Robust Shape Model for Multi-view Car Alignment Yan Li, Leon Gu and Takeo Kanade

43. Outline Problem Introduction. How to do? Discussion.

44. Outline Problem Introduction. How to do? Discussion.

45. Problem Introduction Previous shape alignment model –A hypothesis of Gaussian observation noise. –Use all the observed data to fit a regularized shape. This Gaussian assumption is vulnerable to gross feature detection error. Partial occlusions and spurious background features

46. Outline Problem Introduction. How to do? Discussion.

47. How to do? (1/3) A hypothesis-and-test approach. –Hypothesis: Bayesian Partial Shape Inference (BPSI) algorithm –Test: The hypotheses are then evaluated to find the one that minimizes the shape prediction error.

48. How to do? (2/3) The observed data – –Random sample from Y used to generate hypothesis—shape b and pose θ(s, R, t). used to test hypothesis. Bayesian Partial Shape Inference (BPSI) algorithm –A MAP problem: –A typical missing data problem can be solved by EM.

49. How to do? (3/3) Generate HypothesisTest Hypothesis is the residual between the i-th Corresponding point of

50. Discussion

52. Thank you!