University of Toronto Aug. 11, 2004 Learning the “Epitome” of a Video Sequence Information Processing Workshop 2004 Vincent Cheung Probabilistic and Statistical Inference Group Electrical & Computer Engineering University of Toronto Toronto, Ontario, Canada Advisor: Dr. Brendan J. Frey

Information Processing Workshop 2004 Cheung1 / 12 Outline ●Image epitome ► What? ► Why? ●Implementation computation issues ► Efficiently implementing the learning algorithm ●Video epitome ► Extension to videos ► Video inpainting

Information Processing Workshop 2004 Image Learning Video Cheung2 / 12 Image Epitome ●Jojic, N., Frey, B., & Kannan, A. (2003). Epitomic analysis of appearance and shape. In Proc. IEEE ICCV. ●Miniature, condensed version of the image ●Accurately accounts for the interesting properties of the image ●Applications ► object detection ► texture segmentation ► image retrieval ► compression

Information Processing Workshop 2004 Image Learning Video Cheung3 / 12 Image Epitome Examples

Image Learning Video Information Processing Workshop 2004 Cheung4 / 12 Epitome Input image Training Set Sample Patches Unsupervised Learning Learning the Image Epitome e Z1Z1 Z2Z2 ZMZM … TMTM T2T2 T1T1 Bayesian network e – epitome T k – mapping Z k – image patch

Image Learning Video Information Processing Workshop 2004 Cheung5 / 12 Shifted Cumulative Sum Algorithm

Image Learning Video Information Processing Workshop 2004 Cheung6 / 12 Collecting Sufficient Statistics

Image Learning Video Information Processing Workshop 2004 Cheung7 / 12 Extending Epitomes to Videos ●Desire a miniature, condensed version of a video sequence ●Want it to accurately account for the interesting properties of the video ●Applications ► optic flow ► segmentation ► texture transfer ► layer separation ► compression ► noise reduction ► inpainting

Image Learning Video Information Processing Workshop 2004 Cheung8 / 12 Input Video Frame 1 Frame 2 Frame 3 Training Set Sample Patches Video Epitome Unsupervised Learning Video Epitome

Image Learning Video Information Processing Workshop 2004 Cheung9 / 12 Video Epitome Example Temporally Compressed Spatially Compressed

Image Learning Video Information Processing Workshop 2004 Cheung10 / 12 Video Inpainting (1) ●Fill in missing portions of a video ► damaged films ► occluding objects ●Reconstruct the missing pixels from the video epitome

Image Learning Video Information Processing Workshop 2004 Cheung11 / 12 Video Inpainting (2)

Information Processing Workshop 2004 Cheung12 / 12 Conclusion ●Improved the efficiency of learning image epitomes ●Extended the concept of epitomes to video sequences ●Demonstrated the ability of video epitomes to model motion patterns through video inpainting