1. Learning sparse representations to restore, classify, and sense images and videos Guillermo Sapiro University of Minnesota Supported by NSF, NGA, NIH, ONR, DARPA, ARO, McKnight Foundation

2. Learning Sparsity 2 Martin Duarte Rodriguez Ramirez Lecumberry

3. Learning Sparsity 3 Overview Introduction –Denoising, Demosaicing, Inpainting –Mairal, Elad, Sapiro, IEEE-TIP, January 2008 Learn multiscale dictionaries –Mairal, Elad, Sapiro, SIAM-MMS, April 2008 Sparsity + Self-similarity –Mairal, Bach, Ponce, Sapiro, Zisserman, pre-print. Incoherent dictionaries and universal coding –Ramirez, Lecumberry, Sapiro, June 2009, pre-print Learning to classify –Mairal, Bach, Ponce, Sapiro, Zisserman, CVPR 2008, NIPS 2008 –Rodriguez and Sapiro, pre-print, 2008. Learning to sense sparse signals –Duarte and Sapiro, pre-print, May 2008, IEEE-TIP to appear

4. Learning Sparsity 4 Introduction I: Sparse and Redundant Representations Webster Dictionary: Of few and scattered elements

5. Learning Sparsity 5 Relation to measurements Restoration by Energy Minimization Thomas Bayes 1702 - 1761 Prior or regularization y : Given measurements x : Unknown to be recovered Restoration/representation algorithms are often related to the minimization of an energy function of the form  Bayesian type of approach  What is the prior? What is the image model?

6. Learning Sparsity 6 The Sparseland Model for Images MM K N A fixed Dictionary  Every column in D (dictionary) is a prototype signal (Atom).  The vector  contains very few (say L) non-zeros. A sparse & random vector N

7. Learning Sparsity 7 What Should the Dictionary D Be? D should be chosen such that it sparsifies the representations (for a given task!) Learn D : Multiscale Learning Color Image Examples Task / sensing adapted Internal structure One approach to choose D is from a known set of transforms (Steerable wavelet, Curvelet, Contourlets, Bandlets, …)

8. Learning Sparsity 8 Introduction II: Dictionary Learning

9. Learning Sparsity 9 Each example is a linear combination of atoms from D Measure of Quality for D D  X A Each example has a sparse representation with no more than L atoms Field & Olshausen (‘96) Engan et. al. (‘99) Lewicki & Sejnowski (‘00) Cotter et. al. (‘03) Gribonval et. al. (‘04) Aharon, Elad, & Bruckstein (‘04) Aharon, Elad, & Bruckstein (‘05) Ng et al. (‘07) Mairal, Sapiro, Elad (‘08)

10. Learning Sparsity 10 The K–SVD Algorithm – General D Initialize D Sparse Coding Orthogonal Matching Pursuit (or L1) Dictionary Update Column-by-Column by SVD computation over the relevant examples Aharon, Elad, & Bruckstein (`04) XTXT

11. Learning Sparsity 11 Show me the pictures

12. Learning Sparsity 12 Change the Metric in the OMP

13. Learning Sparsity 13 Non-uniform noise

14. Learning Sparsity 14 Example: Non-uniform noise

15. Learning Sparsity 15 Example: Inpainting

16. Learning Sparsity 16 Example: Demoisaic

17. Learning Sparsity 17 Example: Inpainting

18. Learning Sparsity 18 Not enough fun yet?: Multiscale Dictionaries

19. Learning Sparsity 19 Learned multiscale dictionary

20. Learning Sparsity 20

21. Learning Sparsity 21 Color multiscale dictionaries

22. Learning Sparsity 22 Example

23. Learning Sparsity 23 Video inpainting

24. Extending the Models Learning Sparsity 24

25. Universal Coding and Incoherent Dictionaries Consistent Improved generalization properties Improved active set computation Improved coding speed Improved reconstruction See poster by Ramirez and Lecumberry… Learning Sparsity 25

26. Sparsity + Self-similarity=Group Sparsity Combine the two of the most successful models for images Mairal, Bach. Ponce, Sapiro, Zisserman, pre-print, 2009 Learning Sparsity 26

27. Learning to Classify

28. Learning Sparsity 28 Global Dictionary

29. Learning Sparsity 29 Barbara

30. Learning Sparsity 30 Boat

31. Learning Sparsity 31 Digits

32. Which dictionary? How to learn them? Multiple reconstructive dictionary? (Payre) Single reconstructive dictionary ? (Ng et al, LeCunn et al.) Dictionaries for classification! See also Winn et al., Holub et al., Lasserre et al., Hinton et al. for joint discriminative/generative probabilistic approaches Learning Sparsity 32

33. Learning Sparsity 33 Learning multiple reconstructive and discriminative dictionaries With J. Mairal, F. Bach, J. Ponce, and A. Zisserman, CVPR ’08, NIPS ‘08

34. Learning Sparsity 34 Texture classification

35. Semi-supervised detection learning MIT -- Learning Sparsity 35

36. Learning Sparsity 36 Learning a Single Discriminative and Reconstructive Dictionary Exploit the representation coefficients for classification –Include this in the optimization –Class supervised simultaneous OMP With F. Rodriguez

37. Learning Sparsity 37 Digits images: Robust to noise and occlusions

38. Learning Sparsity 38 Supervised Dictionary Learning With J. Mairal, F. Bach, J. Ponce, and A. Zisserman, NIPS ‘08

39. Learning to Sense Sparse Images

40. Motivation Compressed sensing (Candes &Tao, Donoho, et al.) –Sparsity –Random sampling Universality Stability Shall the sensing be adapted to the data type? –Yes! (Elad, Peyre, Weiss et al., Applebaum et al, this talk). Shall the sensing and dictionary be learned simultaneously? Learning Sparsity 40

41. Some formulas…. Learning Sparsity 41 + “RIP (Identity Gramm Matrix)”

42. Design the dictionary and sensing together Learning Sparsity 42

43. Just Believe the Pictures Learning Sparsity 43

44. Just Believe the Pictures Learning Sparsity 44

45. Just Believe the Pictures Learning Sparsity 45

46. Learning Sparsity 46 Conclusions State-of-the-art denoising results for still (shared with Dabov et al.) and video General Vectorial and multiscale learned dictionaries Dictionaries with internal structure Dictionary learning for classification –See also Szlam and Sapiro, ICML 2009 –See also Carin et al, ICML 2009 Dictionary learning for sensing A lot of work still to be done!

47. Please do not use the wrong dictionaries… 12 M pixel image 7 million patches LARS+online learning: ~8 minutes Mairal, Bach, Ponce, Sapiro, ICML 2009 Learning Sparsity 47

48. Learning Sparsity 48