Webcam-synopsis: Peeking Around the World Young Ki Baik (CV Lab.) (Fri)

Webcam-Synopsis: Peeking Around the World  References Webcam Synopsis : Peeking Around the World Yael Pritch, Alex Rav-Acha, Avital Gutman Shmuel Peleg (ICCV 2007) Non-Chronological Video Synopsis and Indexing Yael Pritch, Alex Rav-Acha, Shmuel Peleg (PAMI 2008) Background Cut J. Sun, W. Zhang, X. Tang, and H. Shum (ECCV 2006)

Webcam-Synopsis: Peeking Around the World  What is Video-Synopsis? Video synopsis is compact video data reducing spatio-temporal redundancy in input video.

Webcam-Synopsis: Peeking Around the World  Demo (Final result)

Webcam-Synopsis: Peeking Around the World  How can we make Video Synopsis? Basic concept Detect activity tubes… in input video I. Put activity tubes… to the S… considering relation between - tubes and background. - tube and other tubes.

Webcam-Synopsis: Peeking Around the World  Synopsis Properties of video synopsis S The video synopsis S should be substantially shorter than the original video I. Maximum “activity” from the original video should appear in the synopsis video. The dynamics of the objects should be preserved in the synopsis video. (for example, regular fast-forward may fail to preserve the dynamics of fast objects.) Visible seams and fragmented objects should be avoided.

Webcam-Synopsis: Peeking Around the World OBJECT-BASED SYNOPSIS

Webcam-Synopsis: Peeking Around the World  Object-based synopsis construction Online Phase This phase is done in real time during video capture. Preparing step for response phase. Response Phase started with user query. This phase may take a few minutes, depending on the activity in the time period of interest.

Webcam-Synopsis: Peeking Around the World  Overall flow

Webcam-Synopsis: Peeking Around the World ONLINE PHASE

Webcam-Synopsis: Peeking Around the World  Video Acquisition Video Volume Data y t x I(x, y, t) : 3D Space Time Volume (x, y) : Spatial coordinates of the image t : Frame index

Webcam-Synopsis: Peeking Around the World  Background Video Construction Temporal Median y t x Sorting by intensity Selecting median value 4 minutes Temporal Window

Webcam-Synopsis: Peeking Around the World  Background Video Construction y t x y t x IB

Webcam-Synopsis: Peeking Around the World  Moving Objects(tubes) Extraction Using Background Cut (ECCV 2006) Using min-cut to get a smooth segmentation of foreground objects. Only considering 2D image (not video).

Webcam-Synopsis: Peeking Around the World  Moving Objects(tubes) Extraction Extracting moving objects We have information for… I(x, y, t) : 3D Space Time Volume B(x, y, t) : 3D Space Time Volume for background Current ~ : Current image = I (x, y, current time) : Current background image = B (x, y, current time)

Webcam-Synopsis: Peeking Around the World  Moving Objects(tubes) Extraction Simplify Object detection problem to… Binary labeling f : Foreground = 1, Background = 0 Obtaining the desirable labeling f … → with Gibbs energy function. Color termContrast term

Webcam-Synopsis: Peeking Around the World  Moving Objects(tubes) Extraction Extracting moving objects Unary term (or color term) Foreground energy Background energy Color differences between image and background

Webcam-Synopsis: Peeking Around the World  Moving Objects(tubes) Extraction Using Background Cut (ECCV 2006) Using min-cut to get a smooth segmentation of foreground objects. Only considering 2D image (not video).

Webcam-Synopsis: Peeking Around the World  Moving Objects(tubes) Extraction Extracting moving objects Binary term (or contrast term) In case of same labeling, energy is decreased. In case of different labeling, energy defends on differences between neighboring intensities.

Webcam-Synopsis: Peeking Around the World  Moving Objects(tubes) Extraction Using min-cut algorithm… - Moving objects are extracted.

Webcam-Synopsis: Peeking Around the World  Moving Objects(tubes) Extraction Extracting moving objects Constructing a mask of all foreground pixels in space-time volume. Applying a 3D morphological dilation on mask. Finally we can obtain activity tubes (or objects). x t Example of tubes

Webcam-Synopsis: Peeking Around the World  Moving Objects(tubes) Extraction Object queue Obtained activity tubes are saved in queue. End of online phase…

Webcam-Synopsis: Peeking Around the World RESPONSE PHASE

Webcam-Synopsis: Peeking Around the World  Response phase User query “I would like to watch in one minute a synopsis of the video from this camera captured during the last hour.” “I would like to watch in five minutes a synopsis of the last week.”… Input video Synopsis When user query occurred, response phase are started.

Webcam-Synopsis: Peeking Around the World  Response phase User query “I would like to watch in one minute a synopsis of the video from this camera captured during the last hour.” “I would like to watch in five minutes a synopsis of the last week.”…

Webcam-Synopsis: Peeking Around the World  Creating time lapse background video Time lapse background video (B out ) The background of the synopsis video It should represent the background changes over time. Day-night transitions, etc. It should represent the background of the activity tubes. Background Video Time lapse BG video Sampling uniformly

Webcam-Synopsis: Peeking Around the World  Creating time lapse background video Time lapse background video A temporal activity histogram H a A uniform temporal histogram H t

Webcam-Synopsis: Peeking Around the World  Creating time lapse background video Time lapse background video A temporal activity histogram H a A uniform temporal histogram H t Interpolating the two histograms H i = H a + (1- λ)H t Background Video (b) Time lapse BG video Sampling with H i B out

Webcam-Synopsis: Peeking Around the World  Selecting tubes and stitching Definition Input video Synopsis : temporal mapping (or time shift) : tube with time segment : shifted tube with time segment

Webcam-Synopsis: Peeking Around the World  Select tubes and stitching Energy for temporal mapping M : Queue : user selected weights : target tube : other tube

Webcam-Synopsis: Peeking Around the World  Select tubes and stitching Graph Synopsis t Longest tube tube Node number = number of tube Label = number of frame t label

Webcam-Synopsis: Peeking Around the World  Select tubes and stitching Energy for temporal mapping M Unary term Activity Cost Synopsis Only pixel that were not entered into the synopsis are added to the activity cost.

Webcam-Synopsis: Peeking Around the World  Select tubes and stitching Energy for temporal mapping M Unary term Consistency with background Synopsis : border of the mapped activity tube

Webcam-Synopsis: Peeking Around the World  Select tubes and stitching Energy for temporal mapping M Binary term Collision Cost Synopsis This expression give a low penalty to pixel whose color is similar to the background…

Webcam-Synopsis: Peeking Around the World  Select tubes and stitching Energy for temporal mapping M Binary term Temporal Consistency Cost Synopsis Preserving the chronological order of events

Webcam-Synopsis: Peeking Around the World  Select tubes and stitching E(M) is minimized by Min-cut algorithm. Stitching the synopsis video To make more reliable result… → The α–Poisson Image Blending End of response phase…

Webcam-Synopsis: Peeking Around the World  Conclusion The method to creating a short video that is a synopsis of and video stream has been presented.  Discussion Contribution Found out new interesting application. Can we find more contribution? Finding higher dimensional problem and Solving it. 3D volume animation, etc.