1. 1 What do color changes reveal about an outdoor scene? KalyanFabianoWojciechToddHanspeter SunkavalliRomeiroMatusikZicklerPfister Harvard University Adobe Systems * * IEEE Conference on Computer Vision and Pattern Recognition 2008

2. 2 Outdoor time-lapse

3. 3 Time-lapse in Computer Vision Geo-location [ Jacobs et al. 2007 ] Intrinsic Images [ Weiss 2001, Matsushita et al. 2003,… ] £ = Time-lapse sequence Texture Time-varying Shading [ Sunkavalli et al. 2007 ] Editing

4. 4 Coherence in time-lapse Most previous work makes use of only the temporal structure in time-lapse. overcast skyclear skysunset Outdoor time-lapse also has a colorimetric structure.

5. 5 Outline 1.A physics-based model. 2.A method to fit the model to data. 3.Applications of the model.

6. 6 Outline 1.A physics-based model. 2.A method to fit the model to data. 3.Applications of the model.

7. 7 A model for outdoor time-lapse Assumption 1 : Static scene

8. 8 A model for outdoor time-lapse ¸ ¸ ¸ observed pixel values scene color under canonical lighting Assumption 2 : Linear Transforms for Re-illumination color transform

9. 9 A model for outdoor time-lapse Assumption 3 : Ambient sky and direct sun illumination sky illumination sun illumination mixing weights

10. 10 A model for outdoor time-lapse Assumption 4 : 2-d subspace for daylight spectra [ Judd et al. 1964, Sastri and Das 1968, …, Hernández-Andrés et al. 2000 ]

11. 11 A model for outdoor time-lapse Assumption 4 : 2-d subspace for daylight spectra

12. 12 A model for outdoor time-lapse illumination coefficients

13. 13 A model for outdoor time-lapse scene color under canonical lighting (normalized)

14. 14 A model for outdoor time-lapse

15. 15 A model for outdoor time-lapse shadows

16. 16 A model for outdoor time-lapse shading

17. 17 A model for outdoor time-lapse 1-d projection of normals onto solar plane

18. 18 Outline 1.A physics-based model. 2.A method to fit the model to data. 3.Applications of the model.

19. 19 Fitting the model albedoshadowsnormals illum. bases angular velocity illum. coeffs. brightness over-constrained system 320 X 240 images, 100 frames 23,040,000 measurements 8,141,220 parameters

20. 20 Fitting the model 1. Estimate albedo and illumination bases 2. Estimate shadows Reconstruct 3. Estimate remaining parameters

21. 21 Fitting results originalreconstruction 240 x 360 images, 120 frames, ~5 mins. interval, 7.36% RMS error

22. 22 Fitting results originalerror x 3 240 x 360 images, 120 frames, ~5 mins. interval, 7.36% RMS error

23. 23 Outline 1.A physics-based model. 2.A method to fit the model to data. 3.Applications of the model.

24. 24 Application I : Color Constancy [ D’Zmura and Lennie 1986, Forsyth 1990, Funt et al. 1991, Finlayson and Funt 1994, Brainard and Freeman 1997, Barnard et al. 1997, Finlayson et al. 2001, Ebner 2004,… ]

25. 25 Application I : Color Constancy originalcolor corrected

26. 26 Application II : Scene Reconstruction

27. 27 Application II : Scene Reconstruction

28. 28 Application II : Scene Reconstruction

29. 29 Application III : Geo-location

30. 30 Application III : Geo-location

31. 31 Summary 1.Outdoor time-lapse sequences are ubiquitous. 2.They are a rich source of scene information. 3.By using both temporal and colorimetric coherence we can access that scene information.

32. 32 Future Work 1.Robust Fitting (mutual illumination, non-lambertian surfaces, foreground clutter). 2.Multiple viewpoints. 3.Estimating weather and atmospheric conditions.

33. 33 Thank you!