1. Hiroyuki Takeda, Hae Jong Seo, Peyman Milanfar EE Department University of California, Santa Cruz Jan 11, 2008 Statistical Image Quality Measures

2. UCSC MDSP Lab OverviewBackground CCA-based Similarity Measure (Full-reference) Slide 1 Conclusion SVD-based Quality Measure (No-reference)

3. UCSC MDSP Lab Develop quantitative measures that automatically predict the perceived image quality Objective Quality Assessment Full-reference No-reference Reduced-reference Applications Image acquisition, compression, communication, displaying, printing, restoration Slide 2

4. UCSC MDSP Lab OverviewBackground CCA-based Similarity Measure (Full-reference) Conclusion SVD-based Quality Measure (No-reference)

5. UCSC MDSP Lab Full-Reference Image Quality Measure Structural Similarity Measure [1] Focus on perceived changes in structural information variation unlike error based approach ( i.e. MSE or PSNR ) MSE : 210 Original image Contrast stretched JPEG compressed Blurred [1] Zhou Wang et al, “ Image Quality Assessment: From Error Visibility to Structural Similarity ”, IEEE TIP ‘ 04 [1] Zhou Wang et al, “ Image Quality Assessment: From Error Visibility to Structural Similarity ”, IEEE TIP ‘ 04 Zhou Wang Zhou Wang Slide 3 Salt-pepper Mean shifted

6. UCSC MDSP Lab Structural Similarity Measure Three components : Luminance, Contrast, Structure Small constant Slide 4 Image patches being compared

7. UCSC MDSP Lab Drawback of SSIM Sensitive to spatial translation, rotation, and scale changes due to simple correlation coefficient A powerful statistical tool : Canonical Correlation Analysis (Hotelling, 1936) Solution OriginalRotation Zoom Out Translation SSIM: 0.549 SSIM: 0.551 SSIM: 0.505 Slide 5

8. UCSC MDSP Lab : canonical correlation New Statistical Image Quality Measure Canonical Correlation Analysis (CCA) : Find out a pair of direction vectors which maximally correlate the two datasets maximally correlate the two datasets : Useful property  Affine–invariance Slide 6

9. UCSC MDSP Lab CCA New Statistical Image Quality Measure Canonical Correlation Structural Similarity Measure GxGy P GxGyP PP GxGx Gy : Local Search Window at i th position Slide 7 P : Pixel intensity G x, G y : Gradients AB

10. UCSC MDSP Lab New Statistical Image Quality Measure Mathematical Solution 1) Calculate Covariance Matrix 2) Solve coupled eigen-value problems 3) Define CCSIM as largest canonical correlation Slide 8

11. UCSC MDSP Lab Examples (1) Original Image Zoom Out Slide 9 12

12. UCSC MDSP Lab Examples (2) CCSIMSSIM 1 0.340.73 2 Zoom Out 21 Original Image Slide 10

13. UCSC MDSP Lab Examples (2) Original Image Translation 13 Slide 11

14. UCSC MDSP Lab Examples (2) CCSIMSSIM 1 0.380.75 3 Original Image Translation 31 Slide 12

15. UCSC MDSP Lab Examples (3) Original Image Rotation 14 Slide 13

16. UCSC MDSP Lab CCSIMSSIM 1 0.410.77 4 Original Image Rotation 41 Slide 14 Examples (3)

17. UCSC MDSP Lab JPEG Compression Example Slide 15 Clean image (QF=100) JPEG(QF=10)JPEG(QF=50) 123 0.899 bits/pixel 0.352 bits/pixel 8 bits/pixel

18. UCSC MDSP Lab 2 1 0.90 0.85 SSIM Clean Image JPEG (QF =50) JPEG (QF =10) Slide 16 JPEG Compression Example 1 2 3 CCSIM 3 1 SSIM 0.79 0.79 CCSIM

19. UCSC MDSP Lab Clean Image VS Compressed Images Slide 17 Quality JPEG quality factor SSIM CCSIM

20. UCSC MDSP Lab Denoising Example Clean Image Denoised by SKR[2] WGN(sigma=15) Slide 18 [2] Takeda et al., “ Kernel Regression for image processing and reconstruction ”, IEEE TIP ‘ 07 123

21. UCSC MDSP Lab Denoising Example CCSIM SSIM Slide 19 Clean Image WGN( sigma =15 ) Denoised by SKR CCSIM SSIM 1 2 3 2 1 3 1 0.47 0.47 0.89 0.89

22. UCSC MDSP Lab Clean VS (Noisy & Denoised images) WGN: Noise level WGN: Noise level Quality Slide 20 Quality WGN: Noise level WGN: Noise level Clean VS Noisy Clean VS Denoised SSIM CCSIM SSIM CCSIM

23. UCSC MDSP Lab Motion Estimation Resolution enhancement from video frames captured by a commercial webcam (3COM Model No.3719) Steering Kernel Regression Super-resolution Slide 21

24. UCSC MDSP Lab Super-resolution Example Clean Image (512 x 512) Super-resolved by SKR Low resolution Sequence (64x64 32 frames) Slide 22 123

25. UCSC MDSP Lab Super-resolution Example SSIM Slide 23 Clean Image Super-resolved by SKR Low resolution Sequence( 32 frames) CCSIM 31 1 2 3 0.870.91

26. UCSC MDSP Lab OverviewBackground CCA-based Similarity Measure (Full-reference) Conclusion SVD-based Quality Measure (No-reference)

27. UCSC MDSP Lab No-Reference SVD-Based Measure SVD N x N Singular value decomposition of local gradient matrix: Local orientation dominance It becomes close to 1 when there is one dominant orientation in a local area. It takes on small values in flat or highly textured (or pure noise) area. So, this quantity tells us about the “edginess” of the region being examined.

28. UCSC MDSP Lab Properties of Local Orientation Dominance(1) Slide 25 Density function for i.i.d. white Gaussian noise N: the window size [1] A. Edelman. Eigenvalues and condition numbers of random matrices, SIAM Journal on Matrix Analysis and Applications 9 (1988), 543-560. Note : the PDF is independent from the noise variance, but depends on the window size. [2] X. Feng and P. Milanfar. Multiscale principal component Analysis for Image Local Orientation Estimation, Proceeding of 36 th Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, Nov. 2002 N=3 N=5 N=7 N=9 N=11

29. UCSC MDSP Lab Properties of Local Orientation Dominance(2) Slide 26 The mean values for a variety of test images with added white Gaussian noise. N = 11 The mean values for pure noise are always constant. Remember the number 0.06

30. UCSC MDSP Lab The Performance Analysis Suppose we have a noisy image and a denoised version using some filter: The residual image is essentially just noise. of the residual image must be close to the value expected of the residual image must be close to the value expected for pure noise. for pure noise. Slide 27 If the filter cleans up the given image effectively, : a given noisy image : the estimated (denoised) image : the residual image

31. UCSC MDSP Lab Example (1) Image denoising by bilateral filter Slide 28 Denoising experiment Bilateral filter has two parameters: Spatial smoothing parameter, and radiometric smoothing parameter The original image A noisy image, Added white Gaussian noise, SNR=20dB, PSNR=29.25dB, RMSE = 8.67 C. Tomasi and R. Manduchi, “ Bilateral filtering for gray and color images ”, Proceedings of the 1998 IEEE International Conference of Computer Vision, Bombay, India, pp. 836-846, January 1998.

32. UCSC MDSP Lab The Performance Analysis of Bilateral Filter Slide 29 N = 11 The plot of as a function of the smoothing parameters:

33. UCSC MDSP Lab Denoising Result Slide 30 Bilateral filter PSNR = 42.87dB, RMSE = 1.833 The noisy image Residual

34. UCSC MDSP Lab The Performance Analysis of Bilateral Filter Slide 31 The plot of as a function of the smoothing parameters: N = 11

35. UCSC MDSP Lab Denoising Result Slide 32 Bilateral filter PSNR = 39.57dB RMSE = 2.68 The noisy image Residual The filter also removes image contents.

36. UCSC MDSP Lab What If We Pick the Parameters by the Best RMSE? Slide 33 The plot of RMSE as a function of the smoothing parameters:

37. UCSC MDSP Lab Slide 34 Denoising Result Bilateral filter, PSNR = 42.87dB RMSE = 1.832 The noisy image Residual

38. UCSC MDSP Lab Slide 35 Example (2) Iterative Steering Kernel Regression The original imageThe noisy image, Added white Gaussian noise, SNR=5.6dB, PSNR = 20.22dB RMSE = 24.87 Iteratively cleaning up noisy images Using the local orientation dominance, we find the optimal number of iterations.

39. UCSC MDSP Lab Slide 36 Denoising Result (1) The plot of as a function of the smoothing parameters:

40. UCSC MDSP Lab Slide 37 Denoising Result ISKR, IT = 15, PSNR = 31.33 dB RMSE = 6.92 The noisy image Residual

41. UCSC MDSP Lab Slide 38 If the Ground Truth is Available, The plot of RMSE as a function of the smoothing parameters: RMSE

42. UCSC MDSP Lab Slide 39 Denoising Result ISKR, IT = 12, PSNR = 31.69 dB RMSE = 6.64 The noisy image Residual

43. UCSC MDSP Lab OverviewBackground CCA-based Similarity Measure (Full-reference) Conclusion SVD-based Quality Measure (No-reference)

44. UCSC MDSP Lab Conclusion Two new statistical quality measures CCSIM(CCA-based) : full-reference Slide 40 SVD-based measure: no-reference CCSIM is a general version of SSIM The proposed methods can be easily extended to video using 3-d local window. We showed examples of JPEG compression, denoising, and super Resolution with comparison to SSIM SVD-based measure is applicable for any denoising filter. We illustrated application to global parameter optimization. Locally adaptive parameter optimization is also possible.

45. UCSC MDSP Lab Authors [1] Hiroyuki Takeda : hiro@soe.ucsc.edu hiro@soe.ucsc.edu www.ucsc.edu/~htakeda [2] Hae Jong Seo : rokaf@soe.ucsc.edu rokaf@soe.ucsc.edu www.ucsc.edu /~rokaf [3] Peyman Milanfar : milanfar@soe.ucsc.edu milanfar@soe.ucsc.edu www.ucsc.edu/~milanfar

46. UCSC MDSP Lab Thank you !

47. UCSC MDSP Lab Super-resolution Example Clean Image Super-resolved by SKR Down-sampled(2)+WGN(sigma=15) Extra 1 123

48. UCSC MDSP Lab Super-resolution Example SSIM Extra 2 Clean Image Super-resolved by SKR Down-sampled(2) +WGN (sigma=15) CCSIM 31 1 2 3 0.710.85