1. Color Computer Vision Martin Jagersand Readings: – Szeliski, 2.3.2 – Forsyth and Ponce, Chapter 6 What is the physics of color? How is color sensed? How is it represented? Applications in vision?

2. Physics: Light = Electro Magnetic Radiation Color = Wavelength

3. Newton 1665 Image from http://micro.magnet.fsu.edu/ White sunlight light: composed of about equal energy in all wavelengths of the visible spectrum Sunlight

5. The Physics of Light Some examples of the spectra of light sources © Stephen E. Palmer, 2002 Rel. power

6. What is the inherent dimensionality of color space ? For a surface Color = response curve

7. For a Human Color is a psychological property of our visual experiences when we look at objects and lights, not a physical property of those objects or lights (S. Palmer, Vision Science: Photons to Phenomenology)Color is a psychological property of our visual experiences when we look at objects and lights, not a physical property of those objects or lights (S. Palmer, Vision Science: Photons to Phenomenology) Color is the result of interaction between physical light in the environment and our visual systemColor is the result of interaction between physical light in the environment and our visual system Wassily Kandinsky (1866-1944), Murnau Street with Women, 1908

9. Biological vision and color constancy One of the most shared images on facebook &TwitterOne of the most shared images on facebook &Twitter

10. Photo receptor distribution in the human eye

11. 100 212 50 42 175 66 244 255 31 0 196 Human RGB Reponses RGBRGB What happens as the intensity of the light changes? 115 47 28 Color perception

12. Color receptors “Red” cone“Green” cone“Blue” cone Response of k’th cone = Principle of univariance: cones give the same kind of response, in different amounts, to different wavelengths. Output of cone is obtained by summing over wavelengths. Responses measured in a variety of ways

13. b( )g( ) r( ) RGB response Response of each R,G or B is integral of incoming light with sensitivity function Response curves are overlapping !

14. Metamers Two different Spectral Energy Distributions with the same RED, GREEN, BLUE response are termed metamers.Two different Spectral Energy Distributions with the same RED, GREEN, BLUE response are termed metamers. b( ) g( ) r( ) Radiance (Energy) Wavelength

15. Color mixing Source: W. Freeman Cartoon spectra for color names:

16. Additive color mixing Colors combine by adding color spectra Light is added to black. Source: W. Freeman + =

17. Examples of additive color systems http://www.jegsworks.com http://www.crtprojectors.co.uk/ CRT phosphors multiple projectors

18. Subtractive color mixing Colors combine by multiplying color spectra. Pigments remove color from incident light (white). Source: W. Freeman - =

19. Why specify color numerically? Accurate color reproduction is commercially valuableAccurate color reproduction is commercially valuable –Many products are identified by color (“golden” arches) Few color names are widely recognized by English speakersFew color names are widely recognized by English speakers –11: black, blue, brown, grey, green, orange, pink, purple, red, white, and yellow. –Other languages have fewer/more. –Common to disagree on appropriate color names. Color reproduction problems increased by prevalence of digital imaging – e.g. digital libraries of art.Color reproduction problems increased by prevalence of digital imaging – e.g. digital libraries of art. –How to ensure that everyone perceives the same color? –What spectral radiances produce the same response from people under simple viewing conditions? Forsyth & Ponce

20. RED GREEN BLUE yellow magenta cyan RGB Standard Color System RGB is a standard system, but is formally defined in terms of the CIE system.

21. Filtering Colors Short wavelength Long wavelength

22. Color Constancy How do humans adapt to the variability of color?

23. Recognizing color differences Color Constancy

24. Color Constancy: Land’s demonstration

25. Land’s Demonstration

26. Invented for color television Backward compatible with B/W TV Y given higher bandwidth than I/Q Other color spaces: YUV or YIQ

27. red green yellow cyan blue magenta INTENSITY hue saturation Other color spaces: HSI

28. I S H HSI space I = ( R + G + B )/3 S = ( 1 - min (R,G,B)/ I ) H = 0 + (G-B)/  if max is R = 1/3 + (B-R)/  if max is G = 2/3 + (R-G)/  if max is B (  is (max-min) of RGB) HSI: Factoring out intensity

29. RGB: sensor-based description HSI: axes better aligned with intrinsic surface reflectance CMY, CMYK: subtractive color, used in printers YIQ : television, YUV: used in JPEG Y I Q RGBRGB.30.59.11.60 -.28 -.32.21 -.52.31 = luminance red - cyan axis magenta - green axis Summary: Other Color Spaces C = 1-R M = 1-G Y = 1-B

30. Homogeneous Region Sample PCA-fitted ellipsoid

31. Homogeneous Color Region: Photometry

32. Chromakeying Artists’ palette Margaret Fleck’s work on image filtering David Forsyth’s work on color constancy Applications

33. Color histograms Color histograms for indexing and retrievalColor histograms for indexing and retrieval Swain and Ballard, Color Indexing, IJCV 1991.Color Indexing uses opponent-axes Tested on product packaging: “The only target for which the effectiveness of Histogram Backprojection suffers badly is that of Charmin paper towels”

34. RGB A bit of binning... feature vectors model Test 1 Test 2 Color Histograms

35. Intersection Distance (can be weighted)  min(M i, T i ) i  Ti Ti i   will be greater than (M i - T i ) ’ W (M i - T i ) 1 0 0 0 0 0 0 0 0 1 0.5 0 0 0 0 0 0 1 0 0 0 0 0 0.5 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 will be smaller than Histogram Metrics

36. Color-based image retrieval Given collection (database) of images:Given collection (database) of images: –Extract and store one color histogram per image Given new query image:Given new query image: –Extract its color histogram –For each database image: –Compute intersection between query histogram and database histogram –Sort intersection values (highest score = most similar) –Rank database items relative to query based on this sorted order

37. Color-based image retrieval Example database

38. Example retrievals Color-based image retrieval

39. Example retrievals Color-based image retrieval

40. model Image Histogram Backprojection  w i min(M i, L i ) i For each local histogram of the appropriate size, give its location a weighted histogram intersection score. Efficiency? Problems? model local histogram from I w i = 1/I i full image's histogram

41. Radiometry for color All definitions are now “per unit wavelength”All definitions are now “per unit wavelength” All units are now “per unit wavelength”All units are now “per unit wavelength” All terms are now “spectral”All terms are now “spectral” Radiance becomes spectral radianceRadiance becomes spectral radiance –watts per square meter per steradian per unit wavelength Radiosity --- spectral radiosityRadiosity --- spectral radiosity

42. RADIANCE VS. LUMINANCE RADIANCE LUMINANCE 1 Watt/(meter^2 x steradian x cos  Lumens at 555nm Luminous Efficacy

43. SPECTRAL ENERGY DISTRIBUTION AND PHOTOPIC LUMINANCE Radiance (Energy) Wavelength I( ) P( ) Human sensitivity curve