Using Photographs to Enhance Videos of a Static Scene Pravin Bhat 1, C. Lawrence Zitnick 2, Noah Snavely 1, Aseem Agarwala 3, Maneesh Agrawala 4, Michael Cohen 1,2, Brian Curless 1, Sing Bing Kang 2 EGSR 2007 University of Washington 1, Microsoft Research Redmond 2 University of California 3, Adobe Systems 4

An overview of Spacetime Fusion

Motivation Low quality video Input Video

Motivation Low quality video Reconstructed video Input VideoReconstructed Video

Motivation Low quality video Reconstructed video –Reconstructed from photos –Good spatial reconstruction –Bad temporal reconstruction Input VideoReconstructed Video

Motivation Spacetime Fusion result Input VideoSpacetime Fusion Result

Motivation Spacetime Fusion result –Spatial properties of reconstruction –Temporal properties of input video Input VideoSpacetime Fusion Result

Define a 3D gradient field Spacetime Fusion

Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video Spacetime Fusion

Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Spacetime Fusion

Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Video frame: t Video frame: t - 1 Spacetime Fusion

Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Video frame: t Video frame: t - 1 GtGt G t (x, y, t) = V(x, y, t) - V(x, y, t - 1) Spacetime Fusion

Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Video frame: t Video frame: t - 1 GtGt G t (x, y, t) = V(x, y, t) - V(x - u, y - v, t - 1) Spacetime Fusion

Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Increases compatibility between temporal gradients and spatial gradients Spacetime Fusion

Define a 3D gradient field –Spatial gradients from reconstruction –Temporal gradients from input video –Key Idea Temporal gradients defined between motion compensated temporal neighbors Increases compatibility between temporal gradients and spatial gradients Integrate the 3D gradient field Spacetime Fusion

Integrating the gradient field Solve linear system: Av = b

Integrating the gradient field Solve linear system: Av = b Constraints: v x, y, t – v x-1, y, t = G x (x, y, t) v x, y, t – v x, y-1, t = G y (x, y, t) v x, y, t – v x-u, y-v, t = G t (x, y, t) Spacetime Fusion

Applications

Enhanced Exposure

Input Video Edit Propagation

User Edits

Edit Propagation User Edits

Edit Propagation User Edits

Edit Propagation User Edits

Edit Propagation User Edits

Edit Propagation

Edited Video Edit Propagation

Super-Resolution

Conclusion Spacetime fusion

Conclusion Spacetime fusion –Combines spatial and temporal gradients from two different sources

Conclusion Spacetime fusion –Combines spatial and temporal gradients from two different sources –Requires motion vectors for temporal source stereo (static scenes) flow (dynamic scenes)

Conclusion Spacetime fusion –Combines spatial and temporal gradients from two different sources –Requires motion vectors for temporal source stereo (static scenes) flow (dynamic scenes) –Major applications Enforcing temporal coherence Transferring lighting information