1. IMAGE RESTORATION AND REALISM MILLIONS OF IMAGES SEMINAR YUVAL RADO

2. IMAGE REALISM What is CG images? How can we tell the difference? 2

3. TODAY’S TOPICS Super – Resolution What is it? How it’s done? Algorithm. Results. CG2REAL The idea behind. Cosegmatation. Color & texture transfer. Results. 3

4. SUPER – RESOLUTION Methods for achieving high-resolution enlargements of pixel-based images. Estimating missing high-resolution detail that isn’t present in the original image, and which we can’t make visible by simple sharpening. 4

5. HOW IT’S DONE? Using learning based approach for enlarging images. In a training set, the algorithm learns the fine details that correspond to different image regions seen at a low-resolution and then uses those learned relationships to predict fine details in other images. 5

6. TRAINING SET GENERATION Low resolution image High resolution image Low resolution enlargement via bilinear interpolation High resolution high pass filter & contrast normalization Low resolution high pass filter & contrast normalization 6

7. LOW RESOLUTION – HIGH RESOLUTION PROBLEM Input Patch Closest image patches from database Corresponding high-resolution patches from database 7

8. HOW CAN WE SOLVE THIS? Markov Network Problem: very long time to calculate, not practical. 8

9. THE BELIEF PROPAGATION Not giving exact results as the Markov Network, but much faster! Still gives good results. Only three or for iterations of the algorithm is enough for getting the results we need. 9

10. THE BELIEF PROPAGATION – CONT. 10

11. FASTEST METHOD – ONE PASS ALGORITHM Based on the belief propagation, there is a faster algorithm that calculates only the high resolution patch compatibilities of neighboring high resolution patches that are already selected, typically the patches above and to the left, in raster-scan order processing. One pass super resolution generates the missing high-frequency content of a zoomed image as a sequence of predictions from local image information. 11

12. ONE PASS ALGORITHM – DIAGRAM 12

13. RESULTS The training set pictures: 13

14. RESULTS – CONT. Original ImageCubic splineOne pass algorithm 14

15. RESULTS – CONT. Cubic splineOriginal ImageOne pass algorithm 15

16. RESULTS – CONT. 16

17. RESULTS – TRAINING SET DEPENDENCY Training set exampleInput imageOne pass algorithm 17

18. RESULTS – FAILURE EXAMPLE Original Image Cubic spline One pass algorithm 18

19. CG2REAL Improving the Realism of Computer Generated Images using a large Collection of Photographs. Computer Generated CG2REAL 19

20. THE IDEA BEHIND? Use Computer Generated image as an input. Look in real photo collection for similar images. Mark the corresponding area in the CG image. Transfer the color and texture from the real image to the CG image. Smooth the edges. 20

21. THE PROCESS 21

22. FINDING SIMILAR IMAGES Ordering the images in pyramid. The key of the pyramid is a combination of two features: The SIFT features of each image. The color in each feature. 22

23. FINDING SIFT FEATURES 1. Scale Space extrema detection a) Construct Scale Space b) Take Difference of Gaussians c) Locate DoG Extrema 2. Key point localization 3. Orientation assignment 4. Build Key point Descriptors 23

24. COSEGMATATION Segmenting the images from the database and the input CG image. Matching similar regions in all images. All in one step! 24

25. COSEGMATATION – CONT. 25

26. COSEGMATATION – CONT. 26

27. COSEGMATATION – RESULTS 27

28. TEXTURE TRANSFER 28

29. TEXTURE TRANSFER – CONT. 29

30. TEXTURE TRANSFER – CONT. 30

31. TEXTURE TRANSFER – CONT. 31

32. TEXTURE TRANSFER – CONT. 32

33. TEXTURE TRANSFER – CONT. After we choose the right label assignment using the function we described earlier we transfer the texture and smooth it nicely to the CG image via Poisson blending. 33

34. COLOR TRANSFER Has two approaches: Color histogram matching. Local color transfer. 34

35. COLOR HISTOGRAM MATCHING Works well between real images. Typically fails when used In matching CG images and real images. This happens because the histogram of CG images is very different the histogram of real. Due to less colors used in CG imaginary. This leads to instability in global color transfer. 35

36. COLOR HISTOGRAM MATCHING 36 CG input Global histogram matching

37. LOCAL COLOR TRANSFER How it’s done? Down sampling of the images. Computation of the color transfer offsets per region from the lower resolution images. smoothing and up sampling the offsets using joint bilateral up sampling. 37

38. LOCAL COLOR TRANSFER - ALGORITHM In each subsampled region that we have we match two histograms: 1D histogram matching on the L* channel. 2D histogram matching on the a* and b* channels. Great results obtained after no more than 10 iterations of this algorithm. 38

39. LOCAL COLOR TRANSFER - RESULTS 39 CG input Color model Local color transfer

40. TONE TRANSFER Decompose the luminance channel of the CG image and one or more real images using a QMF pyramid (QMF - quadrature mirror filter). We apply 1D histogram matching to match the subband statistics of the CG image to the real images in every region. 40

41. TONE TRANSFER – CONT. 41

42. TONE TRANSFER – RESULTS 42 CG inputTone model Local color an tone transfer Close up before Close up after

43. CG2REAL – RESULTS 43 CG Image CG2REAL Image

44. CG2REAL – RESULTS 44 CG Image CG2REAL Image

45. CG2REAL – RESULTS 45 CG Image CG2REAL Image

46. CG2REAL – RESULTS 46 CG Image CG2REAL Image

47. CG2REAL – RESULTS 47 CG Image CG2REAL Image

48. CG2REAL – RESULTS 48 CG Image CG2REAL Image

49. CG2REAL – FAILURES 49

50. CG2REAL – EVALUATION 50

51. CG2REAL – EVALUATION 51

52. THANK YOU FOR LISTENING 52

53. REFERENCES William T. Freeman, Thouis R. Jones, and Egon C. Pasztor IEEE Computer Graphics and Applications 2002, Example-Based Super- Resolution. Johnson MK, Dale K, Avidan S, Pfister H, Freeman WT, Matusik W, CG2Real: Improving the Realism of Computer Generated Images using a Large Collection of Photographs. http://en.wikipedia.org/wiki/Superresolution 53