1. Using Photographs to Enhance Videos of a Static Scene Pravin Bhat 1, C. Lawrence Zitnick 2, Noah Snavely 1, Aseem Agarwala 3, Maneesh Agrawala 4, Michael Cohen 1,2, Brian Curless 1, Sing Bing Kang 2 EGSR 2007 University of Washington 1, Microsoft Research Redmond 2 University of California 3, Adobe Systems 4

2. An overview of Spacetime Fusion

3. Motivation Low quality video Input Video

4. Motivation Low quality video Reconstructed video Input VideoReconstructed Video

5. Motivation Low quality video Reconstructed video –Reconstructed from photos –Good spatial reconstruction –Bad temporal reconstruction Input VideoReconstructed Video

6. Motivation Spacetime Fusion result Input VideoSpacetime Fusion Result

7. Motivation Spacetime Fusion result –Spatial properties of reconstruction –Temporal properties of input video Input VideoSpacetime Fusion Result

8. Define a 3D gradient field Spacetime Fusion

9. Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video Spacetime Fusion

10. Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Spacetime Fusion

11. Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Video frame: t Video frame: t - 1 Spacetime Fusion

12. Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Video frame: t Video frame: t - 1 GtGt G t (x, y, t) = V(x, y, t) - V(x, y, t - 1) Spacetime Fusion

13. Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Video frame: t Video frame: t - 1 GtGt G t (x, y, t) = V(x, y, t) - V(x - u, y - v, t - 1) Spacetime Fusion

14. Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Increases compatibility between temporal gradients and spatial gradients Spacetime Fusion

15. Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Increases compatibility between temporal gradients and spatial gradients Integrate the 3D gradient field Spacetime Fusion

16. Integrating the gradient field Solve linear system: Av = b

17. Integrating the gradient field Solve linear system: Av = b Constraints: v x, y, t – v x-1, y, t = G x (x, y, t) v x, y, t – v x, y-1, t = G y (x, y, t) v x, y, t – v x-u, y-v, t = G t (x, y, t) Spacetime Fusion

18. Applications

19. Enhanced Exposure

20. Input Video Edit Propagation

21. User Edits

22. Edit Propagation User Edits

23. Edit Propagation User Edits

24. Edit Propagation User Edits

25. Edit Propagation User Edits

26. Edit Propagation

27. Edited Video Edit Propagation

28. Super-Resolution

29. Conclusion Spacetime fusion

30. Conclusion Spacetime fusion –Combines spatial and temporal gradients from two different sources

31. Conclusion Spacetime fusion –Combines spatial and temporal gradients from two different sources –Requires motion vectors for temporal source stereo (static scenes) flow (dynamic scenes)

32. Conclusion Spacetime fusion –Combines spatial and temporal gradients from two different sources –Requires motion vectors for temporal source stereo (static scenes) flow (dynamic scenes) –Major applications Enforcing temporal coherence Transferring lighting information