1. Performance Evaluation of Object Detection Algorithms for Video Surveillance
Jacinto C. Nascimento, Member, IEEE, and Jorge S. Marques
IEEE Transactions On Multimedia
VOL.8, NO.4, AUGUST 2006

2. Outline Introduction Related Work Segmentation Algorithms
Proposed Framework
Tests on PETS2001 Dataset
Conclusions

3. Introduction (1/4)
Video surveillance systems rely on the ability to detect moving objects in the video streams.
It should be reliable and effective.
unconstrained environments
non stationary background
different motion patterns…etc

4. Introduction (2/4)
Approaches to characterize the performance of video segmentation:
Pixel based methods
Template based methods
Object based methods
Three major drawback:
Several types of error should be considered.
Some methods are based on the selection with or without persons.
It is not possible to define a unique ground truth.

5. Introduction (3/4)
Five segmentation algorithms are considered as examples and evaluated.
BBS, W4, SGM, MGM, and LOTS.
Several types of errors are considered.
Correct Detections, Detection Failures, Splits, Merges, Splits/Merges, and False Alarms.
Provide segmentation results of these algorithms on the PETS2001 sequence.
We also consider multiple interpretations.

6. Introduction (4/4) Segmentation Algorithms Proposed Framework
(BBS, W4, SGM, MGM, LOTS)
Proposed Framework
(User Friendly Interface)
Segmentation of video images
Create the ground truth
Performance Evaluation
(CD, DF, , Split, Merge, S/M, and FA)

7. Related Work (1/3)
Background subtraction is simple to detect moving objects in video sequences.
by comparing the difference with a threshold
Several difficulties arise when background image is corrupted by noise.
camera movements
fluttering objects (e.g., tree waving)
illumination changes
clouds, shadows

8. Related Work (2/3) Some works use a deterministic background model.
admissible interval for each pixel
maximum rate of change in consecutive images, …etc
Most works rely on statistical models of the background.
Each pixel is a random variable with a probability distribution.
e.g., Pfinder system uses a Gaussian Model.
mixture of Gaussian Models.

9. Related Work (3/3) For shadows and non-stationary backgrounds:
show changes (e.g., sun motion) and rapid changes (clouds, rain, or abrupt changes…etc)
recursively update the background parameters and thresholds
Presence of ghosts
Static objects suddenly starts to move.
Combining background subtraction with frame differencing or by high level operation.

10. Segmentation Algorithms
Basic Background Subtraction W4
detection algorithm used in the W4 system [17]
Single Gaussian Model
Multiple Gaussian Model
Lehigh Omnidirectional Tracking System
Used to detect small non cooperative targets [18]
[17] “W4: real time surveillance of people and their activities”
[18] “Into the woods: Visual surveillance of non-cooperative camouflaged targets in complex outdoor settings”

11. Segmentation Algorithms BBS: Basic Background Subtraction
Computing the difference between the current frame and the background image.
Classify each pixel as foreground region if
For pixels associated with the same object
by connected component analysis
(threshold)
intensity (current frame, 3x1 vector)
mean intensity (background)

12. Segmentation Algorithms Algorithm used in W4 System
modified threshold!
designed for grayscale images
Three features:
Min: minimum intensity
Max: maximum intensity
D: maximum intensity difference between consecutive frames
Classify the pixel I(x,y) as a foreground pixel if
modified!
[17] W4: real-time surveillance of people and their activities

13. Segmentation Algorithms SGM: Single Gaussian Model
The mean and covariance of each pixel:
updated recursively
Classify each pixel as active or background
α: constant, I(x,y): pixel of the current frame (YUV)
If l(x,y) is small, the pixel is classified as active!
[1] Pfinder: real-time tracking of the human body

14. Segmentation Algorithms MGM: Multiple Gaussian Model
MGM models each pixel I(x,y) as a mixture of N (N=3) Gaussian Distributions.
I(x,y) is a 3x1 vector (R,G,B)
The mixture model is dynamically updated.
N Gaussian Distributions with respective weights
weight: non match components of the mixture are not modified
[2] Learning Patterns of Activity Using Real-Time Tracking

15. Segmentation Algorithms LOTS: Lehigh Omnidirectional Tracking System
Use two gray level background images B1, B2.
initialized using a set of T consecutive frames
Targets are detected using two thresholds
high threshold, low threshold
User Specified!

16. Segmentation Algorithms LOTS: Lehigh Omnidirectional Tracking System
Each pixel is considered as active if
A target is a set of connected active pixel that a subset of them verifies:
, high threshold (TH), and low threshold (TL) are updated recursively!
[18] Into the woods: Visual surveillance of non-cooperative camouflaged targets in complex outdoor settings

17. Error Analysis and Classification
Proposed Framework
Principles:
Select a set sequences
1 frame/second
Object Detection
By Automatic Procedure
Manually Correction
User Friendly Interface
Detection Failure, False Alarm…etc
Error Analysis and Classification
Statistics
Performance Evaluation!

18. Proposed Framework Interface used to create ground truth manually.
output of the detector
four active regions
4 false alarms!
User can easily edit it and provide the correct segmentation!

19. Proposed Framework
Compare the output of the algorithm with the ground truth segmentation.
Region Matching
Region Overlap
Area Matching
Multiple Interpretation

20. Proposed Framework Several cases are considered:
Correct Detection (CD): 1-1 match
False Alarm (FA): 0-1 match
Detection Failure (DF): 1-0 match
Merge Region (M): many-1 match
Split Region (S): 1-many match
Split-Merge Region (SM)
Correspondence:
ground truth – detector output

21. Proposed Framework Region Matching
Binary Correspondence Matrix:
Defines correspondence between active regions.

22. Different matching cases:
Correct Detection
Detection Failure!
False Alarm!
Merge
Split
Split-Merge

23. Proposed Framework Region Overlap
Overlap Requirement = 20%
DF! (Overlap < 20%)
CD! (Overlap > 20%)
2 DF! (Overlap < 20%)
Split (Overlap > 20%)

24. Proposed Framework Area Matching
higher percentage of match size, better active regions produced by the algorithm.
For correctly detected regions,
Characterize the performance of the detector!

25. Proposed Framework Multiple Interpretations
Correct Split Example:
Should be considered as valid!
manual segmentation SGM segmentation

26. Proposed Framework Multiple Interpretations
Wrong Split Example:
Wrong Segmentation!
manual segmentation W4 segmentation

27. Proposed Framework Multiple Interpretations
different merged regions groups
Labeling Matrix M
Region Linking Procedure with three objects A, B, C

28. Tests on PETS2001 Dataset
Evaluate several object detection algorithms using PET2001 dataset.
Training (3064 frames) and test sequences (2688 frames) are used.
The first 100 images were used to build the background model.
The algorithm were evaluated using 1 frame/second.
Area of 25 pixel was chosen, and overlap requirement is 10%.
Ground Truth vs. Detector Output

29. Tests on PETS2001 Dataset Choice of the Model Parameter (BBS)
ROC for different value of α : BBS
Performance of BBS is independent of α.
α = α = α = 0.15
T=0.2 is the best value!

30. Tests on PETS2001 Dataset Choice of the Model Parameter (SGM)
ROC for different value of α : SGM
see for -400 < T < -150
seems less DF and FA!
α = α = α = 0.15
Choose α = 0.05, T=-400!

31. Tests on PETS2001 Dataset Choice of the Model Parameter (MGM)
ROC for different value of α : MGM
Performance of MGM is strongly depend on the value of T.
α = α = α = 0.05
Choose α = 0.008, T > 0.9!

32. Tests on PETS2001 Dataset Choice of the Model Parameter (LOTS)
ROC for different background update rate : LOTS
variation of sensitivity from 10% to 110%
Background update at every:
1024th Frame th Frame th Frame

33. Tests on PETS2001 Dataset Performance Evaluation (Case I.)
Performance of five object detection algorithms
If a moving object stops and remains still, it is considered an active region.

34. Tests on PETS2001 Dataset Performance Evaluation (Case II.)
Performance of five object detection algorithms
If a moving object stops and remains still, it is integrated in the background after one minute.

35. Tests on PETS2001 Dataset Complexity vs. Performance
by Appendix
BBS, LOTS, W4, SGM have a similar computational complexity.
MGM requires higher computational cost!
MGM requires higher complexity, but the performance is not as good as the LOTS and SGM.

36. Conclusion
This paper proposes a framework for the evaluation of object detection algorithms.
Detector Output vs. Ground Truth
consider multiple interpretations
Measuring the percentage of each type of error.
The best results were achieved by the LOTS and SGM algorithms.