1. Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech

2. Still Photos

3. Video Clips

4. Video Textures

5. Related work Image-based rendering Novel still views from still images View Morphing [Seitz 96] Lightfields / Lumigraph [Levoy 96, Gortler 96] Façade [Debevec 96] Virtualized Reality [Kanade 97] many more…

6. Related work Texture synthesis Infinite extension of 2D pattern [Heeger 95] [de Bonet 97] [Efros 99]

7. Related work Temporal texture synthesis Video as a spatio-temporal texture Statistical Learning of Multidimensional Textures [Bar-Joseph 99] [Wei 00]

8. Related work Video Rewrite Assemble video snippets to generate lip motion [Bregler 97]

9. Periodic motion analysis Uses similar analysis tools Related work [Niyogi 94] [Seitz 97] [Polana 97] [Cutler 00]

10. Problem Statement video clipvideo texture

11. Approach How do we find good transitions?

12. Finding Good Transitions Compute L 2 distance D i, j between all frames Similar frames make good transitions frame ivs. frame j

13. Markov Chain Representation Similar frames make good transitions

14. Transition Costs Transition from i to j if successor of i is similar to j Cost function: C i  j = D i+1, j

15. Transition Probabilities Probability for transition P i  j inversely related to cost: high  low 

16. Preserving Dynamics

18. Cost for transition i  j C i  j = w k D i+k+1, j+k

19. Preserving Dynamics – Effect Cost for transition i  j C i  j = w k D i+k+1, j+k

20. Deadends No good transition at the end of sequence

21. Future Cost Propagate future transition costs backward Iteratively compute new cost F i  j = C i  j +  min k F j  k

22. Future Cost Propagate future transition costs backward Iteratively compute new cost F i  j = C i  j +  min k F j  k

23. Future Cost Propagate future transition costs backward Iteratively compute new cost F i  j = C i  j +  min k F j  k

24. Future Cost Propagate future transition costs backward Iteratively compute new cost F i  j = C i  j +  min k F j  k

25. Propagate future transition costs backward Iteratively compute new cost F i  j = C i  j +  min k F j  k Q-learning Future Cost

26. Future Cost – Effect

27. Finding Good Loops Alternative to random transitions Precompute a good set of loops up front (using dynamic programming)

28. Visual Discontinuities Problem: Visible “Jumps”

29. Crossfading Solution: Crossfade from one sequence to the other.

30. Morphing Interpolation task:

31. Morphing Interpolation task: Compute correspondence between pixels of all frames

32. Morphing Interpolation task: Compute correspondence between pixels of all frames Interpolate pixel position and color in morphed frame based on [Shum 2000]

33. Results – Crossfading / Morphing

34. Jump Cut Crossfade Morph

35. Crossfading

36. Frequent Jump & Crossfading

37. Video Portrait Useful for web pages

38. Combine with IBR techniques Video Portrait – 3D

39. Region-based Analysis Divide video up into regions Generate a video texture for each region

40. Automatic Region Analysis

41. User selects target frame range User-controlled Video Textures slowvariablefast

42. Lengthen / shorten video without affecting speed Time Warping shorteroriginallonger

43. Video-based animation Like sprites computer games Extract sprites from real video Interactively control desired motion ©1985 Nintendo of America Inc.

44. Video sprite extraction

45. Video sprite control Augmented transition cost:

46. Video sprite control Need future cost computation Precompute future costs for a few angles. Switch between precomputed angles according to user input [GIT-GVU-00-11]

47. Interactive fish

48. Summary Video clips  video textures define Markov process preserve dynamics avoid dead-ends disguise visual discontinuities

49. Summary Extensions regions external constraints video-based animation

50. Discussion Some things are relatively easy

51. Discussion Some are hard

52. Future Work Better analysis of complex motions New optimality metrics More powerful video-based animation Other kinds of video-based rendering

53. A Final Example