1 Real Time, Online Detection of Abandoned Objects in Public Areas Proceedings of the 2006 IEEE International Conference on Robotics and Automation Authors : Nathaniel Bird, Stefan Atev, Nicolas Caramelli, Robert Martin, Osama Masoud, Nikolaos Papanikolopoulos

2 Outline  Introduction  Method Description A. Low-Level Processing B. Short-Term Logic C. Long-Term Logic D. Image Similarity  Results  Conclusions and Future Work

3 I. Introduction  This paper addresses this issue by presenting an algorithm for automated detection of abandoned objects.  Abandoned object : a stationary object has not been touching a person for some time threshold.  The method must Online in real time Stay active around the clock Not detect still people as abandoned objects Detect abandoned objects even if they are occluded by moving crowds of people for periods of time

4 II. Method Description A. Low-Level Processing  Background segmentation is performed using [6].  Processing only every tenth frame is still a high enough rate considering the temporal scale at which the events of interest occur.  [6] C. Stauffer and W. E. L. Grimson, “ Adaptive background mixture models for real-time tracking, ” Proceedings of the IEEE Computer Vision and Pattern Recognition, vol. 2, pp , June 1999

5 II. Method Description A. Low-Level Processing  The background segmentation method is restricted to user-specified regions of interest in the image. block out areas of the image where any background changes detected can be considered as noise (walls) Remove areas that are too far from the camera for accurate abandoned object identification

6 II. Method Description A. Low-Level Processing  Binary foreground mask  Blob extraction is then performed on the binary foreground mask.  Correlating the blobs detected in the last frame with the blobs detected in the current frame.

7 II. Method Description B. Short-Term Logic  Blob types: Abandoned Object (A) Person (P) Still Person (SP) Unknown (U)  Blob behaviors: Creation Splits Merges Blob centroid velocity

8 II. Method Description B. Short-Term Logic  Person Group (PG) A PG is created containing the new blobs that split from a P or an SP. All blobs contained within a PG are classified as U until one of them becomes a P, at which all other blobs within it can be classified normally. This is to stop a sitting person from being incorrectly classified as an abandoned object if they place a bag beside them that splits from their blob.  : threshold velocity above which an abandoned object, still person, or unknown becomes a person.

9 II. Method Description B. Short-Term Logic

10 II. Method Description B. Short-Term Logic

11 II. Method Description C. Long-Term Logic  The long term logic maintains a set of potential abandoned objects and a set of still people. Contour in the image plane Timestamp of when they were first detected.  When an A type blob is found, it first checked if it does not overlap with any item in the potential abandoned object or still person sets. Yes => copied into potential abandoned object set No => ignore  So did SP type blob.

12 II. Method Description C. Long-Term Logic  All items in the potential abandoned object set and the still person set are checked every time a frame is processed. If their corresponding area in the binary foreground mask is filled less than some percentage, p, then the item is dropped. P were empirically found to be between 75% and 80%

13 II. Method Description C. Long-Term Logic  time threshold t If during a check it is discovered that t time has elapsed since an item was added to the potential abandoned object set, an alarm is triggered for that object.  After the alarm is triggered, the long-term logic adjusts a mask used by the background segmentation module so that it will not improperly learn the abandoned object into the background.

14 II. Method Description D. Image Similarity  When the potential abandoned object is first detected, a copy of the image at its location is saved.  At every time step, the area surrounding the object ( a “ halo ” excluding the object) is checked for significant foreground activity.  If there is no noticeable foreground activity in the halo, the current image is compared pixel-by-pixel to the stored image for the potential abandoned object, and an average per-pixel difference is calculated.

15 II. Method Description D. Image Similarity  An exponential running average of this difference is then updated.  If the value of the exponential running average exceeds an empirically determined threshold, the potential abandoned object is deemed to be moving too much to be a stationary object. => reclassified to a still person.

16 III. Results A. Alarm Description  The following information is what is recorded for every alarm : 1. Identification Number 2. Start Time 3. Trigger Time 4. End Time 5. Image-Plane Location

17 III. Results B. Ground Truth  The ground truth for a given video sequence is determined manually for every sequence by a human operator.

18 III. Results C. PED/PAT Score Description  A candidate match is declared if there is sufficient spatial proximity and/or overlap between the two alarms as well as a temporal distance below a specific tolerance.  The candidate matches will usually result in a many-to-many relationship Bipartite graph  We therefore give each edge a weight equal to the timestamp difference between the two alarms and then find the minimum-weighted maximum- cardinality matching.

19 III. Results D. Overall Score Description  We define an overall score as follows:  x is relative importance we wish to give to the PED score over the PAT score. x=0 => PAT plot X=0.5 => weight PED and PAT equally X=1 => PED plot  We use a value of x=0.75 because we consider finding true alarms more important than some false positives.

20 III. Results E. Test Sequences

21 III. Results F. Results

22 III. Results F. Results

23 IV. Conclusions and Future Work  We have presented a method to detect abandoned objects that works online in real time, uses color data, can adapt to scene changes around the clock, does not detect still people as abandoned objects and detects abandoned objects even if they are occluded by moving crowds of people for periods of time.  The results for densely populated scenes are not as good, indicating that future research should look into defining a short- term logic that characterizes the behavior of blobs corresponding to crowds.