1. Adaptive background mixture models for real-time tracking 信息行业化工程中心 赵红

2. introduction Paper information Research background Related work Method Limitation Future work Conclusion

3. Authors and papers Adaptive background mixture models for real-time tracking Computer Vision and Pattern Recognition,1999 IEEE Computer Society Conference Cited by 1826 times Chris Stauffer the Massacusetts Institute of Technology in Electrical Engineering and Computer Science. stauffer@ai.mit.edu http://people.csail.mit.edu/stauffer/Home/index.php W.E.L Grimson: A professor of Computer Science and Engineering at the Massachusetts Institute of Technology A fellow of AAAI, also a fellow of IEEE http://www.csail.mit.edu/user/807

4. Research Background Application: smart surveillance, video index, three-dimensional reconstruction. Motion tracking main goal: given a frame sequence from a fixed camera, detecting all the foreground objects Naive description of the background subtraction: | frame (i)– background (i)| > Th Sometimes the most interesting video sequences involve: a) lighting changes (gradual, sudden( such as cloud ) ), b) scene changes c) and moving objects. high-frequencies background objects (such as tree, branches, sea waves, and similar). oscillations

5. Preparations we consider the values of a particular pixel over time as a “ pixel process”. define: (1) 1

6. Related Work The average the images over time 3 2 9 0 2 3 2 1 0 2 1 2 7 3 [2 3 14 5 7 2 3 4 5 3 4 3 12 2] 5 The background value average Find a threshold 4 Subtract the average result 0 0 0 1 4 5 3 9

7. The average the images over time Update limitation: the noise are accumulation as the time, it is not robust to scenes with many moving objects particularly if they move slowly; It also cannot handle bimodal backgrounds; very memory consuming. Advantage: easy 、 fast (4)

8. shows two scatter plots from the same pixel taken 2 minutes apart 23

9. Related work What is the gaussian distribution? If a random variable with probability density function: it said that X satisfy the Gaussian distribution, Noted: The is the mean value, and the is variance (2) (3) 6

10. 45 (4)shows a bi-model distribution of a pixel values resulting from specularities on the surface of water. (5)shows another bi-modality resulting from monitor flicker

11. Related work Median images over time 2 0 0 2 0 1 0 1 2 0 1 0 9 1 1 2 2 3 3 3 3 3 3 4 4 5 5 12 The background value Sort Wight sort Subtract result 0 0 0 1 1 3 3 5 3 2 3 4 5 3 4 3 12 2 3 3 Threshold 6 5 1 8

12. Median images over time Limitation ： it is not robust to scenes with many moving objects particularly if they move slowly, It also cannot handle bimodal backgrounds ；

13. Related work Pfinder : is a human tracking system. System uses several statistical categories, such as color, shape and so on, in a large observation of the environment to obtain the head and hands 2D spatial signature. Every pixel Foreground if Background else Probability threshold

14. Pfinder system has several successful applications, such as signature speech recognition. limitations: a) system has several major assumptions, if ignoring the fact that few prerequisites, system performance will be reduced; b) Although the system can be small and light the scene, or the slow change in processing, but large or sudden change can not be processed; c) the system can only handle the future there is an object (person) the situation

15. Related work Friedman and Russell: EM A pixel Road color Colors corresponding to vehicles shadow color

16. Advantage: Their attempt to mediate the effect of shadows appears to be somewhat successful, Limitation ： pixels which did not contain these three distributions. For example, pixels may present a single background color or multiple background colors resulting from repetitive motions, shadows, or reflectances

17. Idea If each pixel resulted from a particular surface under particular lighting, a single Gaussian would be sufficient to model the pixel value while accounting for acquisition noise. If only lighting changed over time, a single, adaptive Gaussian per pixel would be sufficient. In practice, multiple surfaces often appear in the view frustum of a particular pixel and the lighting conditions change. Thus, multiple, adaptive Gaussians are necessary. We use a mixture of adaptive Gaussians to approximate this process. Each time the parameters of the Gaussians are updated, the Gaussians are evaluated using a simple heuristic to hypothesize which are most likely to be part of the “background process.” Pixel values that do not match one of the pixel’s “background” Gaussians are grouped using connected components. Finally the connected components are tracked from frame to frame using a multiple hypothesis tracker.

18. Online mixture models The recent history of each pixel, the probability of observing the current pixel value is: K is the number of distributions, w is the weight, and is: (5)

19. Online mixture models We assumes that the red, green, blue pixel values are independent and have the same variances (6) (7)

20. Gmm Initialization kmean 4 5 6 21 8 5 9 4 5 8 2 9 8 7 2 8 9 6 4 8 5 78 3 6 14 4 7 5 4

21. Gmm update Match: a pixel value within 2.5 standard deviations of a distribution (for every pixel) Mismatch: the least probable distribution is replaced with a distribution with the current value as its mean value, an initially high variance, and low prior weight. match (8) 7

22. Gmm update If match,else and the weights is updated ： (9)

23. Gmm update Also the mean and variance are updated. The distribution has been matched (10) (11) (12)

24. Gmm update One of the significant advantages of this method is that when something is allowed to become part of the background, it doesn’t destroy the existing model of the background. The original background color remains in the mixture until it becomes the K most probable and a new color is observed. Therefore, if an object is stationary just long enough to become part of the background and then it moves, the distribution describing the previous background still exists with the same µ and σ, but a lower ω and will be quickly reincorporated into the background.

25. Background Model Estimation First, sorting from the matched distribution This value increases both as a distribution gains more evidence and as the variance decreases

26. Background Model Estimation Then the first B distributions are chosen as the background model, where (13) where T is a measure of the minimum portion of the data that should be accounted for by the background. This takes the “best” distributions until a certain portion, T, of the recent data has been accounted for.

27. Connected Components then, Moving regions can be characterized not only by their position, but size, moments, and other shape information. Not only can these characteristics be useful for later processing and classification, but they can aid in the tracking process.

28. Kalman filter: Kalman ， In essence, trying to use all known information to construct the system state variables, the posterior probability density, that is, with the state of the system state transition models predict a priori probability density, and then using the most recent amendments to the observed value obtained a posteriori probability density.

29. Kalman filter k-1 kk+1 X F B u z x w v Q R H

30. Kalman filter 描述一个动态系统 Kalman (14) (15)

31. Kalman filter 状态的预测 (prediction) 估计协方差的预测 (the covariance prediction) (16) (17)

32. Kalman filter 预测残差 新协方差 Kalman 增益 (18) (19) (20)

33. Kalman filter 状态估计更新 状态协方差估计更新 (21) (22)

34. Demo Time 8 910 111213

37. Limitation The background has a very high frequency variation, this model fail to achieve sensitive detection. how fast the background model adapts to change

38. Future work a ) Adding prediction to each Gaussian b) We are working on activity classification and object classification using literally millions of examples c) Relative values of neighboring pixels and correlations with neighboring pixel’s distributions may be useful in this regard ………

39. conclusion This paper has shown a novel, probabilistic method for background subtraction. It involves modeling each pixel as a separate mixture model. This method deals with slow lighting changes by slowly adapting the values of the Gaussians. It also deals with multi-modal distributions caused by shadows, specularities, swaying branches, computer monitors, and other troublesome features of the real world which are not often mentioned in computer vision. It recovers quickly when background reappears and has a automatic pixel-wise threshold. All these factors have made this tracker an essential part of our activity and object classification research. This system has been successfully used to track people in indoor environments, people and cars in outdoor environments, fish in a tank, ants on a floor, and remote control vehicles in a lab setting. This system achieves our goals of real-time performance over extended periods of time without human intervention.

40. references [1] A Dempster, N. Laird, and D. Rubin. “Maximum likelihood fromincomplete data via the Malgorithm,” Journal of the Royal Statistical Society, 39 (Series B):1-38, 1977. [2] Nir Friedman and Stuart Russell. “Image segmentation in video sequences: A probabilistic approach,” In Proc. of the Thirteenth Conference on Uncertainty in Artificial Intelli-gence(UAI), Aug. 1-3, 1997. [3] B.K.P.Horn. Robot Vision, pp. 66-69, 299-333. The MIT Press, 1986. [4] D. Koller, J. Weber, T. Huang, J. Malik, G. Ogasawara, B. Rao, and S. Russel. “Towards robust automatic tra ffi c scene analysis in real-time.” In Proc. of the International Conference on Pattern Recognition, Israel, November 1994. [5] Christof Ridder, Olaf Munkelt, and Harald Kirchner. “Adaptive Background Estimation and Foreground Detection using Kalman Filtering,” Proceedings of International Conference on recent Advances in Mechatronics,ICRAM’95, UNESCO Chair on Mechatronics, 193-199, 1995. [6] W.E.L. Grimson, Chris Stau ff er, Raquel Romano, and Lily Lee. “Using adaptive tracking to classify and monitor activi-ties in a site,” In Computer Vision andPattern Recognition 1998(CVPR98), Santa Barbara, CA. June 1998. [7] Wren, Christopher R., Ali Azarbayejani, Trevor Darrell, and Alex Pentland. “Pfinder: Real-Time Tracking of the Human Body,” In IEEE Transactions on Pattern Analysis andMachine Intelligence, July 1997, vol 19, no 7, pp.780-785.

41. thanks