1. Patch Based Synthesis for Single Depth Image Super-Resolution (ECCV 2012) Oisin Mac Aodha, Neill Campbell, Arun Nair and Gabriel J. Brostow Presented By: Itzik Ben ShabatJanuary 2014

2. Contents  Problem & Motivation  SR General Overview  Related Work  The Proposed Method  Results  Qualitative  Quantitative  Future Work  Paper review

3. Problem & Motivation  How do we convert a Low Resolution (LR) image to High Resolution (HR) ?  Get a better camera (Sensor)

4. Problem & Motivation  Cant get a better camera? Super Resolve the image! (SR)

5. Problem & Motivation  Now do it in RGB-D ! ! ! PMD CamCube - 200X200 PointGrey BumbleBee 2 - 640x480 at 48fps MS Kinect - 640x480

6. SR General Overview Common approaches:  Take multiple LR images from different angles and reconstruct the additional information (requires multiple images)

7. SR General Overview Common approaches:  Use a LR to HR database (requires a database)  Focus on this approach

8. Related Work  Intensity Images  EbSR [15] - Freeman, W.T., Liu, C.: Example-based super resolution. In: Advances in Markov Random Fields for Vision and Image Processing. MIT Press (2011)  Similar:  Filter input  Normalized Patch matching  Solving minimum energy problem (using BP)  Different  Not Designed for RGB-D images  Matching HR and interpolated LR patches Create LR- HR database Perform bicubic interpolation on input image Desaturate and High- pass filtered Normalize contrast Solve patch minimum energy problem using BP Add back low frequency and color

9. Related Work  EbSR looking closer Ground trouth EbSR Output

10.  Intensity Images  ScSr [17] - Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Transactions on Image Processing (2010)  Similar:  Use patches and minimization problem  Different:  Not designed for RGB-D images  Two dictionaries  Database structure (sparse representation)  Solves 2 minimization problems separately – global and local  No noise reduction implementation Learn 2 dictionaries (LR- HR) For each input patch - Find LR representation Apply representation to HR pairs Solve global optimization problem Related Work

11.  Depth + Intensity Hybrids  Cross Bilateral [1] - Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: CVPR. (2007)  Similar:  Specific for RGB-D  Use bilateral filter  Different:  Requires additional input (destination resolution image)  Doesn’t use patches  Solves a fusion problem

12. Construct Multi- Resolution MRF grid Formulate optimization problem Bilinear filter for y initial guess Solve iteratively Related Work  Depth + Intensity Hybrids  MRF SR [25]- Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: NIPS. (2005)  Similar:  Uses MRF  Different:  Uses multi-resolution MRF  Requires additional input (destination resolution image)  Doesn’t use patches  Solves a fusion problem

13. The Proposed Method  Challenges  Construct database  Noise  “Flying pixels” at discontinuities  Wrong depths for specular or dark materials  Edges – jarring artifacts (different than rgb image)

14. The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

15. The Proposed Method – Database  Constructing the database  Less sources for database construction than rgb images  Considered synthetic Vs. Real datasets  Database uses 30 scenes of 800x800 (scenes flipped left to right) – 5.3 million patches  Pruning to remove redundant patches (planar surfaces) SyntheticLaser Scan

16. The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

17. The Proposed Method - Filtering  Noise Reduction  Assumption – High frequency=noise  A. Bilateral filter on input patches before patch normalization  Edge preserving  Noise reducing  Nonlinear  Weighted average of intensity values from nearby pixels  *Used in Adobe Photoshop “Blur” function  B. Bicubic filter on database HR patches before down-sampling

18. The Proposed Method - Filtering Input After Bilateral filtering

19. The Proposed Method - Filtering  Noise Reduction  Pro – Cleaner image for patching  Con – Some data is lost

20. The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

21. The Proposed Method – Matching Input Depth Image

22. N non overlapping low resolution input patches x i For each x i we wish to find its corresponding high resolution y i Patches are normalized The Proposed Method – Matching Input Depth Image

23. The Proposed Method – Matching High Resolution Database

24. Output Depth Image The Proposed Method – Matching Input Depth Image High Resolution Database

25. Output Depth Image High Resolution Database The Proposed Method – Matching Input Depth Image

26. Output Depth Image High Resolution Database The Proposed Method – Matching

27. Output Depth Image … The Proposed Method – Matching Input Depth Image High Resolution Database

28. Output Depth Image High Resolution Database … The Proposed Method – Matching Input Depth Image

29. Output Depth Image … The Proposed Method – Matching Input Depth Image High Resolution Database

30. Output Depth Image … The Proposed Method – Matching Input Depth Image High Resolution Database

31. Output Depth Image … … The Proposed Method – Matching Input Depth Image High Resolution Database

32. The Proposed Method - Matching  Matching Patches to database  Matching is done between LR patches  Kd tree is used for speeding up the process

33. The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

34. The Proposed Method - Reconstruction  Solving minimum energy problem  Solved using TRW-S algorithm (based on belief propagation)  E d -Unary Potential - Difference between normalized matching LR patches

35. The Proposed Method - Reconstruction  E s -Pairwise Potential - Difference between un-normalized HR patch overlaps

36. The Proposed Method - Reconstruction yiyi yjyj  E s - Pairwise Potential -

37. yiyi yjyj The Proposed Method - Reconstruction  E s - Pairwise Potential -

38. yiyi yjyj The Proposed Method - Reconstruction  E s - Pairwise Potential -

39. yiyi yjyj The Proposed Method - Reconstruction  E s - Pairwise Potential -

40. -- E s =( + ( )2)2 )2)2 yiyi yjyj The Proposed Method - Reconstruction

41.  Normalization – is un-normalized based on the input patch min and max values

42. The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

43. The Proposed Method – Filter Results  Noise reduction  C. Post processing Denoising – Outlier detection and correction using threshold Result Result after denoising input

44. Results - Qualitative  Exp 1:  Used Middleburry stereo dataset  Down-sampled the ground truth (X2,X4)  Reconstructed  Compared RMSE  Exp 2:  3 laser scans  Upsampled by 4  Ground touth comparison  Exp 3:  Use synthetic Vs. real database

45. Results - Qualitative  Proposed method Vs. Other Methods (Exp. 2) Proposed Method

46. Results - Qualitative  Proposed method - Real Vs. Synthetic training data (Exp. 3)

47. Results - Quantitative  Reminder: MRF RS Cross BilateralScSREbSR Upsampling factor Method used RMSE RGB-D image used Method used RGB-D image used

48. Results - Qualitative Results Movie

49. Conclusions  1 st or 2 nd best from intensity based methods  MRF RS, Cross Bilateral are better but require more data  Speed is not realtime compatible  Super resolving moving depth videos  Synthetic data exhibit better results than scanned data for training.

50. Future work  Extend to exploit temporal context (in video)  Exploit context when querying the database  Develop a sensor specific noise model for better results

51. Paper review  Pros  Self contained  Well referenced  Novel approach  Available resources – website, data, code  Good experimental results  Cons  Non chronological order of subjects (training after method, implementation notes after results, etc.)  Supplied code insufficient

52. Questions?

53. References  Some of the slides and images were taken from the original paper presentation, paper and website - http://visual.cs.ucl.ac.uk/pubs/depthSuperRes/ http://visual.cs.ucl.ac.uk/pubs/depthSuperRes/ 

54. Appendix - filters  Bicubic filter -  Interpolate data points on a two dimensional regular grid  Smoother  Fewer interpolation artifacts  Uses 16 pixels (4X4) Bicubic Bilinear Nearest neighbor