Seminar presented by: Tomer Faktor Advanced Topics in Computer Vision (048921) 12/01/2012 SINGLE IMAGE SUPER RESOLUTION

OUTLINE 1 What is Image Super Resolution (SR)? Prior Art Today – Single Image SR:  Using patch recurrence [Glasner et al., ICCV09]  Using sparse representations [Yang et al., CVPR08], [Zeyde et al., LNCS10]

FROM HIGH TO LOW RES. AND BACK 2 Blur (LPF) + Down-sampling Inverse problem

3 Inverse problem – underdetermined Image SR - how to make it determined: WHAT IS IMAGE SR? HowType Set of low res. imagesMulti-Image External database of high-low res. pairs of image patches Example-Based Image model/priorSingle-Image

4 Always: More pixels - according to the scale factor Low res. edges and details - maintained Bonus: Blurred edges - sharper New high res. details missing in the low res. GOALS OF IMAGE SR

MULTI-IMAGE SR “Classical” [Irani91,Capel04,Farisu04] 5 Fusion

6 Example-based SR (Image Hallucination) Algorithm External database of low and high res. image patches [Freeman01, Kim08] EXAMPLE-BASED SR Blur kernel & scale factor

7 Single-Image SR (Scale-Up) Algorithm Blur kernel & scale factor Image model/prior SINGLE IMAGE SR

8 No blur, only down-sampling  Interpolation:  LSI schemes – NN, bilinear, bicubic, etc.  Spatially adaptive and non-linear filters [Zhang08, Mallat10] Blur + down-sampling: InterpolationDeblurring PRIOR ART

9 LS problem: Add a regularization:  Tikhonov regularization, robust statistics, TV, etc.  Sparsity of global transform coefficients  Parametric learned edge model [Fattal07, Sun08] PRIOR ART

D. Glasner, S. Bagon and M. Irani ICCV 2009 SUPER RESOLUTION FROM A SINGLE IMAGE

BASIC ASSUMPTION Patches in a single image tend to redundantly recur many times within and across scales 11

STATISTICS OF PATCH RECURRENCE 12 The impact of SR is mostly expressed here

13 Patch recurrence  A single image is enough for SR Adapt ideas from classical and example-based SR:  SR linear constraints – in-scale patch redundancy instead of multi-image  Correspondence between low-high res. patches - cross-scale redundancy instead of external database SUGGESTED APPROACH Unified framework

EXPLOITING IN-SCALE PATCH REDUNDANCY 14 In the low res. – find K NN for each pixel Compute their subpixel alignment Different weight for each linear SR constraint (patch similarity)

15 EXPLOITING CROSS-SCALE REDUNDANCY Unknown, increasing res. by factor α Known, decreasing res. by factor α Copy Parent NN Parent Copy

ALGORITHMIC SCHEME 16

17 Coarse-to-fine – gradual increase in resolution, for numerical stability, improves results Back-projection [Irani91] – to ensure consistency of the recovered high res. image with the low res Color images – convert RGB to YIQ:  For Y – the suggested approach  For I,Q – bicubic interpolation IMPORTANT IMPLEMENTATION DETAILS

RESULTS – PURELY REPETITIVE STRUCTURE 18 Input low res. image

19 Bicubic Interpolation Suggested Approach

RESULTS – PURELY REPETITIVE STRUCTURE 20 High res. detail Input low res. image Bicubic interp. Only in-scale redundancy In and cross-scale redundancies

RESULTS – TEXT IMAGE 21 Input low res. image Bicubic Interp. Suggested approachGround Truth Small digits recur only in-scale and cannot be recovered

Bicubic Interp. Edge model [Fattal07] RESULTS – NATURAL IMAGE 22 Input low res. image Example-based [Kim08] Suggested Approach

23 Reasonable assumption – validated empirically Very novel – new SR framework, combining two widely used approaches in the SR field Solution technically sound Well written, very nice paper! Many visual results (nice webpage) PAPER EVALUATION

24 Not fully self-contained – almost no details on:  Subpixel alignment  Weighted classical SR constraints  Back-projection No numerical evaluation (PSNR/SSIM) of the results No code available No details on running time PAPER EVALUATION

R. Zeyde, M. Elad and M. Protter Curves & Surfaces, 2010 ON SINGLE IMAGE SCALE-UP USING SPARSE-REPRESENTATIONS

26 SPARSE-LAND PRIOR d m Dictionary n sparse vector signal n Non-zeros Widely-used signal model with numerous applications Efficient algorithms for pursuit and dictionary learning

27 BASIC ASSUMPTIONS Sparse-land prior for image patches Patches in a high-low res. pair have the same sparse representation over a high-low res. dictionary pair Same procedure for all patches Joint training! [Yang08] & [Zeyde10]

28 JOINT REPRESENTATION - JUSTIFICATION Each low res. patch is generated from the high res. using the same LSI operator (matrix): A pair of high-low res. dictionaries with the correct correspondence leads to joint representation of the patches:

29 RESEARCH QUESTIONS Pre-processing? How to train the dictionary pair? What is the training set? How to utilize the dictionary pair for SR? Post-processing? [Yang08] & [Zeyde10]

30 PRE-PROCESSING – LOW RES. Feature Extraction by HPFs Bicubic Interpolation 12 Low Res. Image

31 PRE-PROCESSING – LOW RES. Dimensionality Reduction via PCA Projection onto reduced PCA basis 3 Features for low res. patches

32 PRE-PROCESSING – HIGH RES. - Bicubic Interpolation

33 TRAINING PHASE Training the Dictionary Pair 12

34 TRAINING THE LOW RES. DICTIONARY 1 K-SVD Dictionary Learning

35 TRAINING THE HIGH RES. DICTIONARY 2

36 WHICH TRAINING SET? External set of true high res. image (off-line training):  Generate low res. image  Collect The image itself (bootstrapping):  Input low res. image = “high res.”  Proceed as before…

37 RECONSTRUCTION PHASE Pre-processing Low Res. Image OMP Image Reconstruction SR Image

38 IMAGE RECONSTRUCTION No need to perform back-projection!

39 RESULTS – TEXT IMAGE Input low res. image Bicubic Interp. PSNR=14.68dB PSNR=16.95dB Suggested Approach Ground Truth Dictionary pair learned off-line from another text image!

40 NUMERICAL RESULTS – NATURAL IMAGES Suggested Alg.Yang et al.Bicubic Interp. SSIMPSNRSSIMPSNRSSIMPSNR Barbara Coastguard Face Foreman Lenna Man Monarch Pepper PPT Zebra Average Dictionary pair learned off-line from a set of training images!

Bicubic Interp. Yang et al. Ground Truth Suggested Approach 41 VISUAL RESULTS – NATURAL IMAGES Artifact

42 COMPARING THE TWO APPROACHES Zeyde et al.Glasner et al. Single Image SR Sparse land prior, joint representation for high-low res. Patch recurrenceBasic assumptions OverlapsSubpixel alignmentMulti-patch constraints Learned dictionary-pair Across-scale redundancy High-low res. correspondence Image pre-processing Coarse-to-fine Image post-processing

43 Input low res. image Bicubic Interp. Suggested Off-line Approach Suggested On-line Approach Glasner et al. VISUAL COMPARISON Appears in Zeyde et al.

44 Input low res. image Bicubic Interp. Suggested Off-line Approach Suggested On-line Approach Glasner et al. Doesn’t appear in Zeyde et al.

45 Reasonable assumptions – justified analytically Novel – similar model as Yang et al., new algorithmic framework, improves runtime and image SR quality Solution technically sound Well written and self-contained Code available Performance evaluation – both visual and numerical PAPER EVALUATION

46 Experimental validation – not full:  Comparison to other approaches – main focus on bicubic interp. and Yang et al.  No comparison between on-line and off-line learning of the dictionary pair on the same image Only “good” results are shown, running the code reveals weakness with respect to Glasner et al. PAPER EVALUATION

47 Extending the first approach to video: Space-time SR from a single video [Shahar et al., CVPR11] Merging patch redundancy + sparse representations in the spirit of non-local K-SVD [Mairal et al., ICCV09] Coarse-to-fine also for the second approach – to improve numerical stability and SR quality FUTURE DIRECTIONS