1. CSE 473/573 Computer Vision and Image Processing (CVIP) Ifeoma Nwogu inwogu@buffalo.edu Lecture 6 – Color

2. Schedule Last class – Image formation – photometric properties Today – Color Readings for today: Forsyth and Ponce Chp. 3

3. What is color? Color is a psychological property of our visual experiences when we look at objects and lights, not a physical property of those objects or lights (S. Palmer, Vision Science: Photons to Phenomenology) Color is the result of interaction between physical light in the environment and our visual system

4. Color Human encoding of color Physics of color Color representation White balancing Applications in computer vision

5. Overview of Color Human encoding of color Physics of color Color representation White balancing Applications in computer vision

6. The Eye The human eye is a camera! But what’s the “film”? Lens - changes shape by using ciliary muscles (to focus on objects at different distances) Pupil - the hole (aperture) whose size is controlled by the iris Iris - colored annulus with radial muscles Retina - photoreceptor cells (cones and rods) – “the film” Slide by Steve Seitz

7. The Retina

8. Light sensitive receptors Cones – responsible for color vision – less sensitive – operate in high light Rods – responsible for gray-scale vision (intensity based perception) – highly sensitive – operate at night Rods and cones are non- uniformly distributed on the retina – Fovea - Small region (1 or 2°) at the center of the visual field containing the highest density of cones – and no rods Slide by Steve Seitz pigment molecules

9. Color matching

10. Color matching experiment 1 Source: W. Freeman

11. Color matching experiment 1 p 1 p 2 p 3 Source: W. Freeman

12. Color matching experiment 1 p 1 p 2 p 3 Source: W. Freeman

13. Color matching experiment 1 p 1 p 2 p 3 The primary color amounts needed for a match Source: W. Freeman

14. Color matching experiment 2 Source: W. Freeman

15. Color matching experiment 2 p 1 p 2 p 3 Source: W. Freeman

16. Color matching experiment 2 p 1 p 2 p 3 Source: W. Freeman

17. Color matching experiment 2 p 1 p 2 p 3 We say a “negative” amount of p 2 was needed to make the match, because we added it to the test color’s side. The primary color amounts needed for a match: p 1 p 2 p 3 Source: W. Freeman

18. Trichromacy In color matching experiments, most people can match any given light with three primaries – Primaries must be independent For the same light and same primaries, most people select the same weights – Exception: color blindness Trichromatic color theory – Three numbers seem to be sufficient for encoding color – Dates back to 18 th century (Thomas Young)

19. Grassman’s Laws Color matching appears to be linear If two test lights can be matched with the same set of weights, then they match each other: – Suppose A = u 1 P 1 + u 2 P 2 + u 3 P 3 and B = u 1 P 1 + u 2 P 2 + u 3 P 3. Then A = B. If we mix two test lights, then mixing the matches will match the result: – Suppose A = u 1 P 1 + u 2 P 2 + u 3 P 3 and B = v 1 P 1 + v 2 P 2 + v 3 P 3. Then A + B = (u 1 +v 1 ) P 1 + (u 2 +v 2 ) P 2 + (u 3 +v 3 ) P 3. If we scale the test light, then the matches get scaled by the same amount: – Suppose A = u 1 P 1 + u 2 P 2 + u 3 P 3. Then kA = (ku 1 ) P 1 + (ku 2 ) P 2 + (ku 3 ) P 3.

20. Physiology of color vision © Stephen E. Palmer, 2002 Three kinds of cones: Ratio of L to M to S cones: approx. 10:5:1 Almost no S cones in the center of the fovea S, M and L cones are respectively sensitive to short-, Medium- and long-wavelength lights

21. Interaction of light and surfaces Reflected color is the result of interaction of light source spectrum with surface reflectance

22. Interaction of light and surfaces What is the observed color of any surface under monochromatic light? Olafur Eliasson, Room for one color

23. Overview of Color Human encoding of color Physics of color Color representation White balancing

24. Any source of light can be completely described physically by its spectrum: the amount of energy emitted (per time unit) at each wavelength 400 - 700 nm. © Stephen E. Palmer, 2002 Relative spectral power The physics of light

25. Electromagnetic spectrum Human Luminance Sensitivity Function

26. Some examples of the spectra of light sources © Stephen E. Palmer, 2002 Rel. power Spectra of light sources

27. Source: Popular Mechanics

28. Some examples of the reflectance spectra of surfaces Wavelength (nm) % Light Reflected Red 400 700 Yellow 400 700 Blue 400 700 Purple 400 700 © Stephen E. Palmer, 2002 Reflectance spectra of surfaces

29. Spectra of some real-world surfaces

30. Color Human encoding of color Physics of color Color representation White balancing Applications in computer vision

31. Linear color spaces Defined by a choice of three primaries The coordinates of a color are given by the weights of the primaries used to match it mixing two lights produces colors that lie along a straight line in color space mixing three lights produces colors that lie within the triangle they define in color space

32. Linear color spaces How to compute the weights of the primaries to match any spectral signal? p 1 p 2 p 3 ? Given: a choice of three primaries and a target color signal Find: weights of the primaries needed to match the color signal p 1 p 2 p 3 A = u 1 P 1 + u 2 P 2 + u 3 P 3.

33. Color matching functions How to compute the weights of the primaries to match any spectral signal? Let c(λ) be one of the matching functions, and let t(λ) be the spectrum of the signal. Then the weight of the corresponding primary needed to match t is λ Matching functions, c(λ) Signal to be matched, t(λ)

34. Computing color matches How do we compute the weights that will yield a perceptual match for any test light using a given set of primaries? 1.Select primaries 2.Estimate their color matching functions: observer matches series of monochromatic lights, one at each wavelength 3.Multiply matching functions and test light … … …

35. Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 Slide credit: W. Freeman Color matching functions for a particular set of primaries p 1 = 645.2 nm p 2 = 525.3 nm p 3 = 444.4 nm Rows of matrix C Computing color matches

36. matches Now have matching functions for all monochromatic light sources, so we know how to match a unit of each wavelength. … Arbitrary new spectral signal is a linear combination of the monochromatic sources. t Computing color matches

37. Fig from B. Wandell, 1996 Computing color matches So, given any set of primaries and their associated matching functions ( C ), we can compute weights ( w ) needed on each primary to give a perceptual match to any test light t (spectral signal).

38. Why is computing the color match for any color signal for a given set of primaries useful? – Want the colors in the world, on a monitor, and in a print format to all look the same. – Want to paint a carton of Kodak film with the Kodak yellow color. – Want to match skin color of a person in a photograph printed on an ink jet printer to their true skin color. Adapted from W. Freeman Computing color matches Image credit: pbs.org

39. Linear color spaces: RGB space Primaries are monochromatic lights (for monitors, they correspond to the three types of phosphors) Subtractive matching required for some wavelengths RGB matching functions RGB primaries

40. Linear color spaces: RGB space Single wavelength primaries Good for devices (e.g., phosphors for monitor), but not for perception

41. Linear color spaces: CIE XYZ Established by the commission international d’eclairage (CIE), 1931 Usually projected for 2D visualization as: (x,y) = (X/(X+Y+Z), Y/(X+Y+Z)) The Y parameter corresponds to brightness or luminance of a color Volume of all visible colors in CIE XYZ cone with vertex at origin Intersect the cone with plane X+Y+Z =1 to Get the 2-d CIE XYZ space

42. Linear color spaces: CIE XYZ Matching functions http://en.wikipedia.org/wiki/CIE_1931_color_space

43. Nonlinear color spaces: HSV Perceptually meaningful dimensions: Hue, Saturation, Value (Intensity)

44. Distances in color space Since it is hard to reproduce color exactly, it is important to know whether a color difference would be noticeable to a human viewer. – therefore, compare the significance of small color differences to several people – Humans can determine just noticeable differences, which when plotted form boundaries of indistinguishable colors. – In CIE XY, the properties of these boundary ellipses depend on where in the space they occur – Euclidean distance is therefore a poor indicator of the significance of color differences. ellipses would be more circular in a more uniform space distances in coordinate space would better represent the difference between 2 colors

45. Distances in color space Are distances between points in a color space perceptually meaningful? Not necessarily: CIE XYZ is not a uniform color space, so magnitude of differences in coordinates are poor indicator of color “distance”.

46. Uniform color spaces Unfortunately, differences in x,y coordinates do not reflect perceptual color differences CIE u’v’ is a projective transform of x,y to make the ellipses more uniform McAdam ellipses: Just noticeable differences in color

47. Attempt to correct this limitation by remapping color space so that just- noticeable differences are contained by circles  distances more perceptually meaningful. Examples: – CIE u’v’ – CIE Lab Uniform color spaces CIE XYZ CIE u’v’ CIE LAB is now almost universally the most popular uniform color space

48. Color Human encoding of color Physics of color Color representation White balancing Applications in computer vision

49. White balance When looking at a picture on screen or print, our eyes are adapted to the illuminant of the room, not to that of the scene in the picture When the white balance is not correct, the picture will have an unnatural color “cast” http://www.cambridgeincolour.com/tutorials/white-balance.htm incorrect white balance correct white balance

50. White balance Light SourcesColor Temperature in K Clear Blue Sky10,000 to 15,000 Overcast Sky6,000 to 8,000 Noon Sun and Clear Sky6,500 Sunlight Average5,400 to 6,000 Electronic Flash5,400 to 6,000 Household Lighting2,500 to 3,000 200-watt Bulb2,980 100-watt Bulb2,900 75-watt Bulb2,820 60-watt Bulb2,800 40-watt Bulb2,650 Candle Flame1,200 to 1,500 Related to the concept of color temperature. – a way of measuring the quality of a light source. – unit for measuring this ratio is in degree Kelvin (K). WB is based on the ratio of blue light to red light – green light is ignored. A light with larger Kelvin value has more blue lights than one with value (lower color temperature)

51. White balance Film cameras: – Different types of film or different filters for different illumination conditions Digital cameras: – Automatic white balance – White balance settings corresponding to several common illuminants – Custom white balance using a reference object http://www.cambridgeincolour.com/tutorials/white-balance.htm

52. White balance Von Kries adaptation – Multiply each channel by a gain factor Gray card – Take a picture of a neutral object (white or gray) – Deduce the weight of each channel If the object is recoded as r w, g w, b w, use weights 1/r w, 1/g w, 1/b w

53. White balance Without gray cards: we need to “guess” which pixels correspond to white objects Gray world assumption – The image average r ave, g ave, b ave is gray – Use weights 1/r ave, 1/g ave, 1/b ave Brightest pixel assumption – Highlights usually have the color of the light source – Use weights inversely proportional to the values of the brightest pixels Gamut mapping – Gamut: convex hull of all pixel colors in an image – Find the transformation that matches the gamut of the image to the gamut of a “typical” image under white light Use image statistics, learning techniques

54. White balance by recognition Key idea: for each of the semantic classes present in the image, – compute the illuminant that transforms the pixels assigned to that class – Do this so that the average color of that class matches the average color of the same class in a database of “typical” images – Use learning techniques to represent the distribution of “typical images” J. Van de Weijer, C. Schmid and J. Verbeek, Using High-Level Visual Information for Color Constancy, ICCV 2007.Using High-Level Visual Information for Color Constancy

55. Mixed illumination When there are several types of illuminants in the scene, different reference points will yield different results http://www.cambridgeincolour.com/tutorials/white-balance.htm Reference: moonReference: stone

56. Spatially varying white balance E. Hsu, T. Mertens, S. Paris, S. Avidan, and F. Durand, “Light Mixture Estimation for Spatially Varying White Balance,” SIGGRAPH 2008Light Mixture Estimation for Spatially Varying White Balance InputAlpha mapOutput

57. Given an input image (a), extract dominant material colors using a voting technique (pixels corresponding to white material colors) (b); and brown material colors (c). Estimate the local relative contributions of each light (d). Voting scheme only labels reliable pixels so unreliable pixels, shown in blue, are inferred using an interpolation scheme (e). The mixture information can then be used to compute a visually pleasing white balance (f).

58. Assumptions from that paper There are two illuminant types present in the scene and their colors are known beforehand. Scenes are dominated by only a small number of material colors. In other words, the set of reflectance spectra is sparse. The interaction of light can be described using RGB channels only, instead of requiring full spectra. Surfaces are Lambertian and non-fluorescent, so image color is the product of illumination and reflectance in each channel. Color bleeding due to indirect illumination can be ignored. Lesson: there are still many open problems in vision that are only partially solved

59. Color Human encoding of color Physics of color Color representation White balancing Applications in computer vision

60. Uses of color in computer vision Color histograms for image matching Swain and Ballard, Color Indexing, IJCV 1991.Color Indexing

61. Uses of color in computer vision Image segmentation and retrieval C. Carson, S. Belongie, H. Greenspan, and J. Malik, Blobworld: Image segmentation using Expectation-Maximization and its application to image querying, ICVIS 1999.

62. Uses of color in computer vision Color-based image retrieval Given collection (database) of images: – Extract and store one color histogram per image Given new query image: – Extract its color histogram – For each database image: Compute intersection between query histogram and database histogram – Sort intersection values (highest score = most similar) – Rank database items relative to query based on this sorted order

63. Example database Color-based image retrieval

64. Example retrievals Color-based image retrieval

65. Example retrievals Color-based image retrieval

66. Uses of color in computer vision Skin detection M. Jones and J. Rehg, Statistical Color Models with Application to Skin Detection, IJCV 2002.Statistical Color Models with Application to Skin Detection

67. Uses of color in computer vision Building appearance models for tracking D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.Tracking People by Learning their Appearance

68. Next class Linear filters Readings for next lecture: – Forsyth and Ponce Chp 4.1, 4.2; Szeliski 3.1-3.3 (optional) Readings for today: – Forsyth and Ponce 3; Szeliski 2.3.2 (optional)

69. Questions