1. Seminar presented by: Tomer Faktor Advanced Topics in Computer Vision (048921) 12/01/2012 SINGLE IMAGE SUPER RESOLUTION

2. OUTLINE 1 What is Image Super Resolution (SR)? Prior Art Today – Single Image SR:  Using patch recurrence [Glasner et al., ICCV09]  Using sparse representations [Yang et al., CVPR08], [Zeyde et al., LNCS10]

3. FROM HIGH TO LOW RES. AND BACK 2 Blur (LPF) + Down-sampling Inverse problem

4. 3 Inverse problem – underdetermined Image SR - how to make it determined: WHAT IS IMAGE SR? HowType Set of low res. imagesMulti-Image External database of high-low res. pairs of image patches Example-Based Image model/priorSingle-Image

5. 4 Always: More pixels - according to the scale factor Low res. edges and details - maintained Bonus: Blurred edges - sharper New high res. details missing in the low res. GOALS OF IMAGE SR

6. MULTI-IMAGE SR “Classical” [Irani91,Capel04,Farisu04] 5 Fusion

7. 6 Example-based SR (Image Hallucination) Algorithm External database of low and high res. image patches [Freeman01, Kim08] EXAMPLE-BASED SR Blur kernel & scale factor

8. 7 Single-Image SR (Scale-Up) Algorithm Blur kernel & scale factor Image model/prior SINGLE IMAGE SR

9. 8 No blur, only down-sampling  Interpolation:  LSI schemes – NN, bilinear, bicubic, etc.  Spatially adaptive and non-linear filters [Zhang08, Mallat10] Blur + down-sampling: InterpolationDeblurring PRIOR ART

10. 9 LS problem: Add a regularization:  Tikhonov regularization, robust statistics, TV, etc.  Sparsity of global transform coefficients  Parametric learned edge model [Fattal07, Sun08] PRIOR ART

11. D. Glasner, S. Bagon and M. Irani ICCV 2009 SUPER RESOLUTION FROM A SINGLE IMAGE

12. BASIC ASSUMPTION Patches in a single image tend to redundantly recur many times within and across scales 11

13. STATISTICS OF PATCH RECURRENCE 12 The impact of SR is mostly expressed here

14. 13 Patch recurrence  A single image is enough for SR Adapt ideas from classical and example-based SR:  SR linear constraints – in-scale patch redundancy instead of multi-image  Correspondence between low-high res. patches - cross-scale redundancy instead of external database SUGGESTED APPROACH Unified framework

15. EXPLOITING IN-SCALE PATCH REDUNDANCY 14 In the low res. – find K NN for each pixel Compute their subpixel alignment Different weight for each linear SR constraint (patch similarity)

16. 15 EXPLOITING CROSS-SCALE REDUNDANCY Unknown, increasing res. by factor α Known, decreasing res. by factor α Copy Parent NN Parent Copy

17. ALGORITHMIC SCHEME 16

18. 17 Coarse-to-fine – gradual increase in resolution, for numerical stability, improves results Back-projection [Irani91] – to ensure consistency of the recovered high res. image with the low res Color images – convert RGB to YIQ:  For Y – the suggested approach  For I,Q – bicubic interpolation IMPORTANT IMPLEMENTATION DETAILS

19. RESULTS – PURELY REPETITIVE STRUCTURE 18 Input low res. image

20. 19 Bicubic Interpolation Suggested Approach

21. RESULTS – PURELY REPETITIVE STRUCTURE 20 High res. detail Input low res. image Bicubic interp. Only in-scale redundancy In and cross-scale redundancies

22. RESULTS – TEXT IMAGE 21 Input low res. image Bicubic Interp. Suggested approachGround Truth Small digits recur only in-scale and cannot be recovered

23. Bicubic Interp. Edge model [Fattal07] RESULTS – NATURAL IMAGE 22 Input low res. image Example-based [Kim08] Suggested Approach

24. 23 Reasonable assumption – validated empirically Very novel – new SR framework, combining two widely used approaches in the SR field Solution technically sound Well written, very nice paper! Many visual results (nice webpage) PAPER EVALUATION

25. 24 Not fully self-contained – almost no details on:  Subpixel alignment  Weighted classical SR constraints  Back-projection No numerical evaluation (PSNR/SSIM) of the results No code available No details on running time PAPER EVALUATION

26. R. Zeyde, M. Elad and M. Protter Curves & Surfaces, 2010 ON SINGLE IMAGE SCALE-UP USING SPARSE-REPRESENTATIONS

27. 26 SPARSE-LAND PRIOR d m Dictionary n sparse vector signal n Non-zeros Widely-used signal model with numerous applications Efficient algorithms for pursuit and dictionary learning

28. 27 BASIC ASSUMPTIONS Sparse-land prior for image patches Patches in a high-low res. pair have the same sparse representation over a high-low res. dictionary pair Same procedure for all patches Joint training! [Yang08] & [Zeyde10]

29. 28 JOINT REPRESENTATION - JUSTIFICATION Each low res. patch is generated from the high res. using the same LSI operator (matrix): A pair of high-low res. dictionaries with the correct correspondence leads to joint representation of the patches:

30. 29 RESEARCH QUESTIONS Pre-processing? How to train the dictionary pair? What is the training set? How to utilize the dictionary pair for SR? Post-processing? [Yang08] & [Zeyde10]

31. 30 PRE-PROCESSING – LOW RES. Feature Extraction by HPFs Bicubic Interpolation 12 Low Res. Image

32. 31 PRE-PROCESSING – LOW RES. Dimensionality Reduction via PCA Projection onto reduced PCA basis 3 Features for low res. patches

33. 32 PRE-PROCESSING – HIGH RES. - Bicubic Interpolation

34. 33 TRAINING PHASE Training the Dictionary Pair 12

35. 34 TRAINING THE LOW RES. DICTIONARY 1 K-SVD Dictionary Learning

36. 35 TRAINING THE HIGH RES. DICTIONARY 2

37. 36 WHICH TRAINING SET? External set of true high res. image (off-line training):  Generate low res. image  Collect The image itself (bootstrapping):  Input low res. image = “high res.”  Proceed as before…

38. 37 RECONSTRUCTION PHASE Pre-processing Low Res. Image OMP Image Reconstruction SR Image

39. 38 IMAGE RECONSTRUCTION No need to perform back-projection!

40. 39 RESULTS – TEXT IMAGE Input low res. image Bicubic Interp. PSNR=14.68dB PSNR=16.95dB Suggested Approach Ground Truth Dictionary pair learned off-line from another text image!

41. 40 NUMERICAL RESULTS – NATURAL IMAGES Suggested Alg.Yang et al.Bicubic Interp. SSIMPSNRSSIMPSNRSSIMPSNR 0.7826.770.7626.390.7526.24Barbara 0.6627.120.6427.020.6126.55Coastguard 0.8233.520.8033.110.8032.82Face 0.9333.190.9132.040.9131.18Foreman 0.8833.000.8632.640.8631.68Lenna 0.7927.910.7727.760.7527.00Man 0.9431.120.9330.710.9229.43Monarch 0.8934.050.8733.330.8732.39Pepper 0.9125.220.8924.980.8723.71PPT3 0.8428.520.8327.950.7926.63Zebra 0.8530.040.8329.590.8128.76Average Dictionary pair learned off-line from a set of training images!

42. Bicubic Interp. Yang et al. Ground Truth Suggested Approach 41 VISUAL RESULTS – NATURAL IMAGES Artifact

43. 42 COMPARING THE TWO APPROACHES Zeyde et al.Glasner et al. Single Image SR Sparse land prior, joint representation for high-low res. Patch recurrenceBasic assumptions OverlapsSubpixel alignmentMulti-patch constraints Learned dictionary-pair Across-scale redundancy High-low res. correspondence Image pre-processing Coarse-to-fine Image post-processing

44. 43 Input low res. image Bicubic Interp. Suggested Off-line Approach Suggested On-line Approach Glasner et al. VISUAL COMPARISON Appears in Zeyde et al.

45. 44 Input low res. image Bicubic Interp. Suggested Off-line Approach Suggested On-line Approach Glasner et al. Doesn’t appear in Zeyde et al.

46. 45 Reasonable assumptions – justified analytically Novel – similar model as Yang et al., new algorithmic framework, improves runtime and image SR quality Solution technically sound Well written and self-contained Code available Performance evaluation – both visual and numerical PAPER EVALUATION

47. 46 Experimental validation – not full:  Comparison to other approaches – main focus on bicubic interp. and Yang et al.  No comparison between on-line and off-line learning of the dictionary pair on the same image Only “good” results are shown, running the code reveals weakness with respect to Glasner et al. PAPER EVALUATION

48. 47 Extending the first approach to video: Space-time SR from a single video [Shahar et al., CVPR11] Merging patch redundancy + sparse representations in the spirit of non-local K-SVD [Mairal et al., ICCV09] Coarse-to-fine also for the second approach – to improve numerical stability and SR quality FUTURE DIRECTIONS