1. Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TAs: Clément Godard & Fabrizio Pece

2. Today’s schedule Computational Time-Lapse Video – Bennett & McMillan, Siggraph 2007 Factored Time-Lapse Video – Sunkavalli, Matusik, Pfister, Rusinkiewicz Siggraph 2007 Video Synopsis & Indexing, – Pritch, Rav-Acha, Peleg, ICCV 2007

3. Time-lapse Photography Definition: when frames are captured at a lower rate than the rate at which they will ultimately be played back. Compare to – Slow-motion – Bullet-time

4. Time-lapse Filmmaker’s Challenges Lighting changes (strobing) High frequency motion (missed action) – Saturation & exposure Camera motion: intentional or not Triggering / data storage / camera safety Useful? Editable?

5. Does the old definition of time-lapse still hold?

6. (Quick Overview of) Computational Time-Lapse Video Bennett & McMillan Siggraph 2007 (Project web page) (though link seems to have died, so see my archive here)here

7. Virtual Shutter

9. Sampling in 1D Uniform Interpolate linearly+ piece-wise vs. Same # of samples, but where needed! Optimum polygonal approximation of digitized curves, Perez & Vidal, PRL’94 – Use Dynamic Programming to find global min.

10. Video

11. Completely User-controllable: Min-Change, Min-Error, Median, etc

12. Factored Time-Lapse Video Sunkavalli, Matusik, Pfister, Rusinkiewicz SIGGRAPH 2007 Project web page

13. Remember Intrinsic Images from Video? Weiss ICCV’01

14. Outdoors: More Than Just Color vs Illumination? OriginalNo Shadows Given: Daytime, under clear-sky conditions Wishlist Remove shadows Render new shadows Change albedo Change global illumination Wishlist Remove shadows Render new shadows Change albedo Change global illumination

15. Outdoors: More Than Just Color vs Illumination? OriginalNo Shadows Lighting due to sun? sky? Surface Normals?

16. Formulation size: width x height x time F : xyt volume of frames over time I sky : Accumulated intensity due to sky-light I sun : Accumulated intensity due to sun-light S sun : Binary; is a pixel out of shadow?

17. Intuition

19. Separate Sky first Photoshop! – Just to pick “non-surfaces” When is sun’s contribution == 0? – Leaving just sky’s contribution

20. “Definitely” shadow: median of darkest 20%, then threshold at 1.5x

21. Bilateral Filter “Definitely” Shadow

23. How Did Skylight Change? I sky : Accumulated intensity due to sky-light W sky : Sky-light image H sky : Sky-light basis curve (1D function!!) – Whole scene got brighter/darker together

24. Blue: skylight curve (same one)

25. Factorization Decompose appearance into – per-pixel W – H curve Alternating Constrained Least Squares (ACLS): – H (t) is held fixed while W is optimized using least squares, then vice versa

26. ACLS: Alternating Constrained Least Squares Inverse shade trees for non-parametric material representation and editing, – Lawrence et al., Siggraph 2006, code online Lawrence et al., Siggraph 2006online

28. Original and I sky

29. I sky and Reconstruction

30. So Far - Explained Sky

31. Now Decompose Sun’s Contributions I sun : Accumulated intensity due to sun-light W sun : Sun-light image, per-pixel weights H sun : Sun-light basis curve : Shift-map; per pixel offset in time

33. I sun : Accumulated intensity due to sun-light W sun : Sun-light image, per-pixel weights H sun : Sun-light basis curve : Shift-map; per pixel offset in time

34. I sun : Accumulated intensity due to sun-light W sun : Sun-light image, per-pixel weights H sun : Sun-light basis curve : Shift-map; per pixel offset in time

35. Total sunlight contribution Sunlight image (like on moon) Shift mapSunlight

36. Factorize Again! For each image, subtract off I sky (t) S sun : Sky-light mask, serves as C, confidence map ACLS again – But add 3 rd update phase, searching for that minimizes reconstruction error of scaled + offset H(t) vs. F(t)

37. Green: sunlight curve (same one, but time-shifted)

38. I sun (t) W sun (time offset)

39. Reconstruction

40. Does It Work? (black curves are calculated sunlight basis curves)

41. Video: FactoredTimeLapseV

42. What have we gained? Can – Remove shadows – Change albedo – Change amount of global illumination Compression Render new shadows: – We have 1 component of the per-pixel Normals, N

43. Estimated Normals (1D)

44. Manipulating Normals

45. (View Results of) Video Synopsis & Indexing Pritch, Rav-Acha, Peleg ICCV 2007, PAMI 2008 (Project web page)

46. What to do with blob-tracks? (Project web page)

47. HDR Time-Lapse Essentially started in 2006 (manually at first)manually at first More are popping up online More