1. Video Segmentation Prepared By NID@L M. Alburbar Supervised By: Mr. Nael Abu Ras University of Palestine Interactive Multimedia Application Development I

2. Contents: o Introduction. o Video. o Segmentation in Video. o Segmentation using Motion. o Motion detection o Image differencing o Background subtraction o Advanced background subtraction o Very advanced background subtraction 2

3. Introduction Video segmentation is different from segmentation of a single image. While several correct solutions may exist for segmenting a single image, there needs to be a consistency among segmentations of each frame for video segmentation. 3

4. Videos o Videos are Image Sequences over Time y x t 25 Images/s. An image is a function: At each time step t we have an image Frame rate = the number of images per second. 4

5. 5 Segmentation in Video o Finding the object(s) o Preprocessing, segmentation Knowledge base Problem domain Image acquisition Preprocessing Segmentation Representation and description Recognition and Interpretation Result

6. 6 Segmentation o Separation of Foreground (object) and Background (everything else = noise) o Result could be a o Binary image o Containing foreground only o Useful for further processing, such as using silhouettes الظل, etc. o Approach o Motion

7. 7 Segmentation using Motion o Assume that only the object is moving => motion can be used to find the object o Motion detection o Image differencing o Background subtraction o Advanced background subtraction o Very advanced background subtraction

8. 8 Image Differencing

9. 9 o The motion in an image can be found by subtracting the current image from the previous image o Algorithm 1.Save image in last frame. 2.Capture current camera image. 3.Subtract image (= difference = motion). 4.Threshold. 5.Delete noise.

10. 10 Subtract Image o Compute pixel-wise o Subtract previous image from input image: o Usually the absolute distance is applied 1.Save image in last frame 2.Capture camera image 3.Subtract image 4.Threshold 5.Delete noise

11. 11 Threshold o Decide, when a pixel is supposed to be considered as a background pixel, or when it is to be considered as a foreground pixel: o Pixel is foreground pixel, if o Pixel is background pixel, if o Problem: What TH?!? 1.Save image in last frame 2.Capture camera image 3.Subtract image 4.Threshold 5.Delete noise

12. 12 Deleting Noise o Single pixels are likely to appear: o Pixel-noise!! o Apply Median filter: o Depending on filter size, bigger spots can be erased o Alternative: Morphology 1.Save image in last frame 2.Capture camera image 3.Subtract image 4.Threshold 5.Delete noise (show: patch: diff)

13. 13 Background Subtraction

14. 14 Background Subtraction o Foreground is moving, background is stable o Algorithm 1.Capture image containing background 2.Capture camera image 3.Subtract image (difference = motion) 4.Threshold 5.Delete noise

15. Advanced Background Subtraction 15

16. 16 Advanced Background Subtraction o What if we have small motion in the background? o Bushed, leaves, etc. and noise in the camera/lighting o (show histo patch) o Learn(!) the background o Capture N images and calculate the average background image (no object present) 1.Calculate average background image 2.Capture camera image 3.Subtract image (= motion) 4.Threshold 5.Delete noise

17. 17 Very Advanced Background Subtraction

18. 18 Very Advanced Background Subtraction o Learn the background and its variations!! o Gaussian models (mean,var) for each pixel!!! o The more images you train on the better!! o Idea: o Some pixels may vary more than other pixels o Algorithm: o Consider each pixel (x,y) in the input image and check, how much it varies with respect to the mean and variance of the learned Gaussian models? 1.Calculate mean and variance for each pixel 2.Capture camera image 3.Subtract image (= motion) 4.Threshold according to variance 5.Delete noise

19. 19 Threshold According to Variance o Threshold can be chosen depending on the variance o A local threshold o Standard Deviation o For example: o If Th min object pixel o Th min = mean –  o Th max = mean +  1.Calculate mean and variance for each pixel 2.Capture camera image 3.Subtract image (= motion) 4.Threshold according to variance 5.Delete noise

20. 20 Conclusion o Motion segmentation o Image differencing (two images) o Background subtraction (one bg. image) o Advanced background subtraction (many bg. images) o Very advanced background subtraction (learn each pixel)

21. References: o http://www.wisdom.weizmann.ac.il/~bagon/slides/ Shai_Tal.ppt http://www.wisdom.weizmann.ac.il/~bagon/slides/ Shai_Tal.ppt o http://www.cvmt.dk/education/teaching/e07/MED 3/IP/IP8-video.ppt http://www.cvmt.dk/education/teaching/e07/MED 3/IP/IP8-video.ppt o http://citeseerx.ist.psu.edu/viewdoc/download?doi =10.1.1.37.7579&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi =10.1.1.37.7579&rep=rep1&type=pdf o http://www.download- it.org/free_files/Pages%20from%20Chapter%206- 1f21e0d0b871a64f220ccdd97ed6070c.pdf http://www.download- it.org/free_files/Pages%20from%20Chapter%206- 1f21e0d0b871a64f220ccdd97ed6070c.pdf 21

22. THANKS FOR LESTENING 22