11/01/06Jean-François Lalonde Natural Color Statistics p. 1 Natural color statistics Jean-François Lalonde Misc-read, November 1 st 2006

11/01/06Jean-François Lalonde Natural Color Statistics p. 2 Image formation & capture BRDF Irradiance Sensor's response Radiance

11/01/06Jean-François Lalonde Natural Color Statistics p. 3 Color spaces RGB  Used in displays HSV  Separates luminance from chroma from “purity” CIE L*a*b*  Separates luminance from chroma  Close to perceptual uniformity R G B

11/01/06Jean-François Lalonde Natural Color Statistics p. 4 Big Picture Is color really important?

11/01/06Jean-François Lalonde Natural Color Statistics p. 5 Is color important? Colors contribute to recognition when they are diagnostic of a scene category [Oliva & Synchs, 2000] Diagnostic Colors Mediate Scene Recognition DiagnosticNon-diagnostic

11/01/06Jean-François Lalonde Natural Color Statistics p. 6 Experiment 3 – phase 1

11/01/06Jean-François Lalonde Natural Color Statistics p. 7 Experiment 3 – phase 1

11/01/06Jean-François Lalonde Natural Color Statistics p. 8 Experiment #2 – phase 2

11/01/06Jean-François Lalonde Natural Color Statistics p. 9 Experiment #2 – phase 2

11/01/06Jean-François Lalonde Natural Color Statistics p. 10 Color is important! 3 Experiments  Colors contribute to recognition when they are diagnostic of a scene category  Faster verification of category membership of scenes when properly colored  Addition of colors to coarse luminance blobs enhance categorization [Olivia & Torralba, 2006] Building the Gist of a Scene: The Role of Global Image Features in Recognition

11/01/06Jean-François Lalonde Natural Color Statistics p. 11 Big Picture – (bis) Is color really important? Can natural color be compactly represented? [Oliva & Synchs, 2000] [Oliva & Torralba, 2006] YES! important for scene recognition

11/01/06Jean-François Lalonde Natural Color Statistics p channels? Human cones: L,M,S color receptors

11/01/06Jean-François Lalonde Natural Color Statistics p channels Low dimensionality of natural reflectances  PCA on spectral data over visible range  98% of energy can be represented using 3 components [Chiao & Cronin, 2000] Coral reefForest

11/01/06Jean-François Lalonde Natural Color Statistics p. 14 Big Picture – revisited Can natural color be compactly represented? [Chiao & Cronin, 2000] How?? [Oliva & Synchs, 2000] [Oliva & Torralba, 2006] YES! important for scene recognition 3 channels account for 98% variability Is color really important?

11/01/06Jean-François Lalonde Natural Color Statistics p. 15 Mapping to know illuminant Color constancy: reduction of the effect of the scene illumination  Recover color under known illuminant Not true object reflectance!  Estimate mapping from unknown  known Linear model  C known = A  C unknown

11/01/06Jean-François Lalonde Natural Color Statistics p. 16 Color constancy setup Known (canonical) illuminant Unknown scene Unknown illuminant

11/01/06Jean-François Lalonde Natural Color Statistics p. 17 Classic paper: Gamut mapping Gamut: convex set [Forsyth, 1990], [Barnard, 1998] Known illuminant Canonical gamut Unknown illuminant a b c B A C aAaA aCaC aBaB Unknown gamut Canonical gamut

11/01/06Jean-François Lalonde Natural Color Statistics p. 18 Gamut mapping: transformations cBcB cAcA cCcC bAbA bCbC bBbB aBaB aCaC aAaA [Forsyth, 1990], [Barnard, 1998], [Finlayson, 1995] Possible to do it in chromaticity space (2-D)

11/01/06Jean-François Lalonde Natural Color Statistics p. 19 G.D. Finlayson ~170 papers on color constancy  Somewhat incremental Color by Correlation  Colors in an image provide information about the illuminant [Finlayson et al., 2001], [Schaefer et al., 2005]

11/01/06Jean-François Lalonde Natural Color Statistics p. 20 Color by Correlation  RGB  chromaticity (no luminance)  Characterize illuminants  Matrix M(c, l) = P(chromaticity c | light l ) 3. Quantize input image 4. Compute correlation 5. Select best illuminant  Max correlation [Finlayson et al., 2001], [Schaefer et al., 2005] Gamut mapping performs similar to using P(c|l)  {0,1}

11/01/06Jean-François Lalonde Natural Color Statistics p. 21 Color constancy Major problem: require calibrated illuminants! Alternative: color flows  How color commonly change together under natural illuminant variation Allow non-linear transformations Data-driven [Miller & Tieu, 2001]

11/01/06Jean-François Lalonde Natural Color Statistics p. 22 Color flows Kernel density estimator Partially-observed color flow Full color flow Slice of RGB cube

11/01/06Jean-François Lalonde Natural Color Statistics p. 23 Color eigenflows Subsample RGB cube Apply PCA, keep first k eigenflows First 3 eigenflows: Original image [Miller & Tieu, 2001]

11/01/06Jean-François Lalonde Natural Color Statistics p. 24 Big Picture – yet again Can natural color be compactly represented? [Chiao & Cronin, 2000] How?? [Barnard, 1998] [Forsyth, 1990] [Finlayson, 1995] [Finlayson et al., 2001] [Miller & Tieu, 2001] [Schaefer et al., 2005] [Oliva & Synchs, 2000] [Oliva & Torralba, 2006] YES! important for scene recognition 3 channels account for 98% variability Mapping to illuminant (color constancy) Is color really important?

11/01/06Jean-François Lalonde Natural Color Statistics p. 25 Color harmony Harmonic colors  Aesthetically pleasing in terms of human visual perception Matsuda's regions [Matsuda, 1995], [Tokumaru et al., 2002],[Cohen-Or et al., 2006]

11/01/06Jean-François Lalonde Natural Color Statistics p. 26 Color harmonization Find best-fitting template Squash all colors inside template (hue only) [Cohen-Or et al., 2006]

11/01/06Jean-François Lalonde Natural Color Statistics p. 27 Color harmonization [Cohen-Or et al., 2006] Before Harmonizing background with foreground Does not change natural images!

11/01/06Jean-François Lalonde Natural Color Statistics p. 28 Color harmonization

11/01/06Jean-François Lalonde Natural Color Statistics p. 29 Big Picture – last time I promise Can natural color be compactly represented? [Chiao & Cronin, 2000] How?? [Barnard, 1998] [Forsyth, 1990] [Finlayson, 1995] [Finlayson et al., 2001] [Miller & Tieu, 2001] [Schaefer et al., 2005] [Matsuda, 1995] [Tokumaru et al., 2002] [Cohen-Or et al., 2006] [Oliva & Synchs, 2000] [Oliva & Torralba, 2006] YES! important for scene recognition 3 channels account for 98% variability Mapping to illuminant (color constancy) Fixed hue intervals (color harmony) Is color really important?

11/01/06Jean-François Lalonde Natural Color Statistics p. 30 Back to psychophysics Observation:  H,S,V variation  retinal stimuli = highly non-linear! Hypothesis:  Retinal stimuli is more sensible to more likely colors  Probabilistic approach [Long et al., 2006]

11/01/06Jean-François Lalonde Natural Color Statistics p. 31 Back to psychophysics [Long et al., 2006] 1600 natural images

11/01/06Jean-François Lalonde Natural Color Statistics p. 32 Our neurons match the data! [Long et al., 2006]

11/01/06Jean-François Lalonde Natural Color Statistics p. 33 I was just kidding, but now I’m really done Can natural color be compactly represented? [Chiao & Cronin, 2000] How?? [Barnard, 1998] [Forsyth, 1990] [Finlayson, 1995] [Finlayson et al., 2001] [Miller & Tieu, 2001] [Schaefer et al., 2005] [Matsuda, 1995] [Tokumaru et al., 2002] [Cohen-Or et al., 2006] [Long et al., 2006] [Oliva & Synchs, 2000] [Oliva & Torralba, 2006] YES! important for scene recognition 3 channels account for 98% variability Mapping to illuminant (color constancy) Fixed hue intervals (color harmony) Probabilistic? Is color really important?

11/01/06Jean-François Lalonde Natural Color Statistics p. 34 Conclusion What about spatial information?  See website...  s/colourPerception/colourPerception.html

11/01/06Jean-François Lalonde Natural Color Statistics p. 35 Thank you! Your reward: You’ve just read ~10 papers in 1 hour!

11/01/06Jean-François Lalonde Natural Color Statistics p. 36 References  [Barnard, 1998] K. Barnard. Color constancy overview, 1998  [Chiao & Cronin, 2000] C.-C. Chiao and T. W. Cronin. Color signals in natural scenes: characteristics of reflectance spectra and effects of natural illumination. J. Opt. Soc. Am. A, 17(2),  [Cohen-Or et al., 2006] D. Cohen-Or, O. Sorkine, R. Gal, T. Leyvand, and Y.-Q. Xu. Color Harmonization. SIGGRAPH, 2006  [Finlayson, 1995] G.D. Finlayson. Coefficient color constancy. Ph.D. Thesis, Simon Fraser University, School of Computing, 1995  [Finlayson et al., 2001] G.D. Finlayson, S. Hordley and P. Hubel. Color by correlation: A simple, unifying framework for color constancy, PAMI 23(11) 2001  [Forsyth, 1990] D. A. Forsyth. A novel algorithm for color constancy. IJCV, 5(1):5-36, 1990  [Long et al., 2006] F. Long, Z. Yand, and D. Purves. Spectral statistics in natural scenes predict hue, saturation and brightness. Proc. Natl. Acad. Sci, 103(15), 2006  [Matsuda, 1995] Y. Matsuda. Color design, Asakura Shoten, 1995  [Miller & Tieu, 2001] E. Learned-Miller and K. Tieu. Color Eigenflows: Statistical Modeling of Joint Color Changes. ICCV, 2001  [Oliva & Synchs, 2000] A. Oliva and P.G. Schyns. Diagnostic colors mediate scene recognition. Cognitive Psychology, 41, 2000  [Oliva & Torralba, 2006] A. Oliva and A. Torralba. Building the Gist of a Scene: The Role of Global Image Features in Recognition. Progress in Brain Research: Visual perception, 155, 23-36, 2006  [Schaefer et al., 2005] G. Schaefer, S. Hordley, and G. Finlayson. A combined physical and statistical approach to colour constancy. CVPR, 2005  [Tokumaru et al., 2002] M. Tokumaru, N. Muranaka, and S. Imanishi. Color design support sustem considering color harmony. International Conference on Fuzzy Systems, 2002.