1. 1 Color Image Processing in the block DCT Space Jayanta Mukhopadhyay Department of Computer Science & Engineering Indian Institute of Technology, Kharagpur, 721302, India jay@cse.iitkgp.ernet.in

2. 2 What is COLOR? Selective emission/reflectance of different wavelengths

3. 3 Color Spectrum IlluminationReflectance Spectrum: Intensity as a function of wavelength.

4. 4 Color Stimuli IlluminationReflectance The colour of an object: is the product of the spectrum of the incident light with the light absorption and/or reflection properties of the object.

5. 5 What is Perceived Color? The response generated by a stimulus in the cones gives the perceived color Three responses

6. 6 Human color perception For human eye – Approximately 65% of all cone are sensitive to red light – 33% are sensitive to green light – 2% are sensitive to blue light But blue cones are the most sensitive.

7. 7 Tri-stimulus Values Integration over wavelength X = ∫C(λ)x(λ) dλ = Σ C(λ)x(λ) Y = ∫C(λ)y(λ) dλ = Σ C(λ)y(λ) Z = ∫C(λ)z(λ) dλ = Σ C(λ)z(λ) Real colors span a subset of the XYZ space. Two different stimuli can have same XYZ values. –Metameris Additive color mixtures modeled by addition in XYZ space.

8. 8 Amounts of three primaries needed to match all wavelengths of the spectrum The curves represented by the cone’s reception are not simple peaks. They are, instead, quite complex curves. They even go negative! RGB is not capable of reproducing every single color we can see.

9. 9 Perceived Color Features Intensity – Sum of the spectrum – Energy under the spectrum Hue – Mean wavelength of the spectrum – What wavelength sensation is dominant? Saturation – Standard deviation of the spectrum – How much achromatic/gray component? Chrominance – Hue and saturation

10. 10 Limitation of Tri-Stimulus Model No physical feel as to how colors are arranged. How do brightness change? How does hue change? Subtractive like paint cannot be modeled by XYZ space.

11. 11 CIE XYZ Space Intensity (I) – X+Y+Z Chrominance (x,y) - (X/I, Y/I) –Chromaticity chart – Projection on a plane with normal (1,1,1) –Reduction of dimension –Similar to 3D to 2D in geometry – Each vector from (0,0,0) is an iso- chrominance line – Each vector maps to a point in the chromaticity chart

12. 12 RGB-to-XYZ Space

13. 13 CIE Chromaticity Chart Shows all the visible colors Achromatic Colors are at (0.33,0.33) – Called white point The saturated colors at the boundary – Spectral Colors

14. 14 Chromaticity Chart: Hue All colors on straight line from white point to a boundary has the same spectral hue – Dominant wavelength

15. 15 Chromaticity Chart: Saturation Purity (Saturation) – How far shifted towards the spectral color – Ratio of a/b – Purity =1 implies spectral color with maximum saturation

16. 16 Color Reproducibility Only a subset of the 3D CIE XYZ space called 3D color gamut Projection of the 3D color gamut –Triangle –2D color gamut Large if using more saturated primaries Cannot describe brightness range reproducibility

17. 17 Standard Color Gamut

18. 18 Color spaces RGB (CIE), RnGnBn (TV – NTSC) XYZ (CIE) UVW (UCS de la CIE), U*V*W* (UCS modified by the CIE) YUV, YIQ, YCbCr HSV, HLS, IHS Munsel colour space (cylindrical representation) CIELuv CIELab

19. 19 RGB-to-YCbCr

20. 20 Color Enhancement

21. 21 Color Processing in the Compressed Domain Computation with reduced storage. Avoid overhead of inverse and forward transform.. Exploit spectral factorization for improving the quality of result and speed of computation.

22. 22 Basic Approaches Modify the DC coefficient for increasing brightness. Aghaglzadeh and Ersoy (1992), Opt.Engg. Modify AC coefficients for increasing contrast. Tang, Peli and Acton (2003), IEEE SPL A combination of both. S. Lee (2007), IEEE CSVT Preserve also colors by processing DCT of chromatic components.

23. 23 Different methods Multi-Contrast Enhancement with Dynamic Range Compression ( S. Lee (2007), IEEE CSVT ) Modification of DC coefficients and AC coefficients (following similar strategy of multi-contrast enhancement). Normalized DC coefficients (x) are modified as follows:

24. 24 Proposed Approach Adjust background illumination. Use DC coefficients of the Y component. Preserve Local Contrast. Scale AC coefficients of the Y component appropriately. Preserve Colors. Preserve Color Vectors in the DCT domain. DCT coefficients of Cb and Cr components.

25. 25 Let μ and σ denote the mean and standard deviation of an image. Contrast ζ of an image is defined here as:. Contrast : Definition Weber Law: where is the difference in luminance between a stimulus and its surround, and L is the luminance of the surround

26. 26 Theorem on Contrast Preservation in the DCT Domain Let  d be the scale factor for the DC coefficient and  a  a be the scale factor for the AC coefficients of a DCT block Y. The processed DCT block Y e is given by: The contrast of the processed image then becomes  a /  d times of the contrast of the original image. In this algorithm  d =  a =  for preservation of the contrast.

27. 27 Preservation of Colours in the DCT Domain Let U and V be the DCT coefficients of the C b and C r components, respectively. If the luminance component Y of an image is uniformly scaled by a factor , the colors of the processed image with Y e, U e and V e are preserved by the following operations:

28. 28 Enhancement by Scaling Coefficients Find the scale factor by mapping the DC coefficient with a monotonically increasing function. Apply scaling to all other coefficients in all the components. For blocks having greater details, apply block decomposition and re-composition strategy.

29. 29 Mapping functions for adjusting the local background illumination (TW) Mitra and Yu, CVGIP’87 (DRC) Lee, CSVT’07 (SF) De, TENCON’89

30. 30 Monotonic Mapping Functions

31. 31 Scaling only DC coefficients

32. 32 Scaling both DC and AC coefficients

33. 33 Preservation of Contrast and Color original

34. 34 Block Decompos. Smaller DCT blocks 8x8 block Apply CES on smaller blocks Block Composition Enhanced Block Enhancement of Blocks with more details

35. 35 Removal of Blocking Artifacts original

36. 36 Some Results original ARMCE MCEDRC TW-CES-BLK MSR

37. 37 Enhancement near Edges ARMCEMCEDRC TW-CES-BLKDRC-CES-BLK SF-CES-BLK

38. 38 Some Results original AR MCE MCEDRC TW-CES-BLK MSR

39. 39 Enhancement near edges ARMCEMCEDRC TW-CES-BLK DRC-CES-BLKSF-CES-BLK

40. 40 Some Results original ARMCE MCEDRCTW-CES-BLK MSR

41. 41 Enhancement near edges AR MCE MCEDRC TW-CES-BLKDRC-CES-BLKSF-CES-BLK

42. 42 Metrics for Comparison Wang and Bovic (SPL, 2002) JPEG Quality Metric (JPQM) Wang and Bovic (ICIP,2002) Susstrunk and Winkler (SPIE, 2004)

43. 43 Approaches under consideration Alpha Rooting (AR) : Aghaglzadeh and Ersoy (1992), Opt.Engg. Multi-Contrast Enhancement (MCE): Tang, Peli and Acton (2003), IEEE SPL Multi-Contrast Enhancement with Dynamic Range Compression (MCEDRC): S. Lee (2007), IEEE CSVT Contrast Enhancement by Scaling (CES): Proposed work Multi-Scale Retinex (MSR) (a reference spatial domain technique): Jobson, Rahman and Woodell (1997), IEEE IP

44. 44 Average Performance Measures TechniquesJPQMCEF Y- QM C b - QM C r - QM AR 8.580.970.800.67 MCE 7.000.940.760.67 MCEDRC 7.920.970.860.67 TW-CES-BLK 7.791.500.900.820.81 DRC-CES- BLK 8.161.180.860.76 SF-CES-BLK 8.131.250.890.780.77

45. 45 Computational Complexities TechniquesPer Pixel Operations AR 1E + 1M MCE 2.19M+1.97A MCEDRC 0.03E+3.97M+2A TW-CES 0.02E+4.02M+1.05A DRC-CES 0.05E+4M+1.08A SF-CES 0.03E+4.02M+1.06A MSR 18E+1866378M+8156703A aE+bM+cA implies a Exponentiation, b Multiplication and c Addition operations.

46. 46 Iterative Enhancement original Iteration no.=1 Iteration no.=2 Iteration no.=3 Iteration no.=4

47. 47 Problem of Color Constancy Three factors of image formation: Objects present in the scene. Spectral Energy of Light Sources. Spectral Sensitivity of sensors. Spectral Power DistributionSurface Reflectance Spectrum Spectral Response of a Sensor

48. 48 Same Scene Captured under Different Illumination Can we transfer colors from one illumination to another one?

49. 49 Computation of Color Constancy Deriving an illumination independent representation. - Estimation of SPD of Light Source. Color Correction - Diagonal Correction. E(λ) To perform this computation with DCT coefficients.

50. 50 Different Spatial Domain Approaches Gray World Assumption (Buchsbaum (1980), Gershon et al. (1988)) ≡ White World Assumption (Land (1977)) ≡

51. 51 Select from a set of Canonical Illuminants  Observe distribution of points in 2-D Chromatic Space.  Assign SPD of the nearest illuminant. Gamut Mapping Approach (Forsyth (1990), Finlayson (1996)) - Existence of chromatic points. Color by Correlation (Finlayson et. al. (2001)) - Relative strength over the distribution. Nearest Neighbor Approach (Proposed) - Mean and Covariance Matrix. - Use of Mahalanobis Distance.

52. 52 Processing in the Compressed Domain Consists of non-overlapping DCT blocks (of 8 x 8). Use DC coefficients of each block. The color space used is Y-Cb-Cr instead of RGB. Chromatic Space for Statistical Techniques is the Cb-Cr space.

53. 53 Different Algorithms under consideration

54. 54 List of Illuminants

55. 55 Images Captured at Different Illumination Source: http://www.cs.sfu.ca/ colour/data.

56. 56 Performance Metrics Estimated SPD: E= True SPD: T=

57. 57 Average Δθ

58. 58 Average Δ rg

59. 59 Average Δ RGB

60. 60 Average Δ L

61. 61 Average Performance over the canonical set.

62. 62 Average Δ θ

63. 63 Average Δ rg

64. 64 Average Δ RGB

65. 65 Average Δ L

66. 66 n l : number of illuminants. n c: size of the 2-D chromaticity space n: number of image pixels f: Fraction of chromaticity space covered. aM+bA  a number of Multiplications and b number of Additions. Time and Storage Complexities

67. 67 Time and Storage Complexities

68. 68 Equivalent No. of Additions per pixel (1 M= 3 A) n=512, n c =32, n l =12, f=1

69. 69 Color Correction: An Example Target Ref. Image (syl-50mr16q) Image captured with (solux-4100) MXW-DCT-YCOR COR-DCT

70. 70 Color Restoration OriginalEnhanced w/o Color Correction Enhanced with Color Correction

71. 71 Conclusion-I Color-constancy computation in the compressed domain : - requires less time and storage. - comparable quality of results. Both NN and NN-DCT perform well compared to other existing statistical approaches. Color constancy computation is useful in restoration of colors.

72. 72 Direct filtering in the 8x8 block DCT space using convolution multiplication properties. Approximate and exact computations by block DCT composition and decomposition. Demonstration of its applications in removing blocking artifacts and image enhancement. Conclusion-II

73. 73 Thanks