University of Toronto Oct. 18, 2004 Modelling Motion Patterns with Video Epitomes Machine Learning Group Meeting University of Toronto Oct. 18, 2004 Vincent Cheung Probabilistic and Statistical Inference Group Electrical & Computer Engineering University of Toronto Toronto, Ontario, Canada Advisor: Dr. Brendan J. Frey

ML Group Meeting, Oct. 18, 2004 Cheung1 / 18 Outline ●Image epitome ► What? ► Why? ► How? ●Implementation computation issues ► Efficiently implementing the learning algorithm ●Video epitome ► Extension to videos ► Filling-in missing information

ML Group Meeting, Oct. 18, 2004 Image Learning Video Cheung2 / 18 Image Epitome ●Jojic, N., Frey, B., & Kannan, A. (2003). Epitomic analysis of appearance and shape. In Proc. IEEE ICCV. ●Miniature, condensed version of the image ●Models the image’s shape and textural components. ●Applications ► object detection ► texture segmentation ► image retrieval ► compression

ML Group Meeting, Oct. 18, 2004 Image Learning Video Cheung3 / 18 Image Epitome Examples

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung4 / 18 Learning the Image Epitome (1)

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung5 / 18 Learning the Image Epitome (2) Epitome Input image Training Set Sample Patches Unsupervised Learning e Z1Z1 Z2Z2 ZMZM … TMTM T2T2 T1T1 Bayesian network e – epitome T k – mapping Z k – image patch E: M:

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung6 / 18 Shifted Cumulative Sum Algorithm

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung7 / 18 Collecting Sufficient Statistics

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung8 / 18 Extending Epitomes to Videos ●Desire a miniature, condensed version of a video sequence ●Want the epitome to model the basic shape, textural, and motion patterns of the video ●Applications ► optic flow ► segmentation ► texture transfer ► layer separation ► compression ► noise reduction ► fill-in / inpainting

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung9 / 18 Input Video Frame 1 Frame 2 Frame 3 Training Set Sample Patches Video Epitome Unsupervised Learning Video Epitome

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung10 / 18 Video Epitome Example Temporally Compressed Spatially Compressed

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung11 / 18 Video Inpainting (1) ●Fill in missing portions of a video ► damaged films ► occluding objects ●Reconstruct the missing pixels from the video epitome

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung12 / 18 Video Inpainting (2)

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung13 / 18 Filling-in Missing Data Likelihood Joint Variational Approx E-Step M-Step

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung14 / 18 Missing Channels ●Generalization of the video inpainting problem ●Inpainting ► Missing entire pixels ●Missing Channels ► Missing one or more of the red, green, or blue (RGB) components of a given pixel ●Epitome must consolidate multiple patches together to piece together the missing channel information ► No training patch contains all the channel information ► Use the epitome to fill-in the missing data

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung15 / 18 Image Missing Channels Fill-in

Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung16 / 18 Video Missing Channels Fill-in

ML Group Meeting, Oct. 18, 2004 Cheung17 / 18 Conclusion ●Improved the efficiency of learning image epitomes ●Extended the concept of epitomes to video sequences ●Demonstrated the ability of video epitomes to model motion patterns through video inpainting

ML Group Meeting, Oct. 18, 2004 Cheung18 / 18 Future Work ●Determining the size of the epitome ► Dependent on the complexity of the image / video ■Minimum description length ■Variational Bayesian ●Optimal patch size(s) ► Problem specific ●Additional transformations into the epitome ► Rotation ► Scale ●Additional video epitome applications ► Super-resolution ► Layer separation ► Object recognition