1. Moving Object Detection and Tracking for Intelligent Outdoor Surveillance Assoc. Prof. Dr. Kanappan Palaniappan palaniappank@missouri.edupalaniappank@missouri.edu Dr. Filiz Bunyak bunyak@missouri.edubunyak@missouri.edu Dr. Sumit Nath naths@missouri.edunaths@missouri.edu Department of Computer Science University of Missouri-Columbia

2. Visual Surveillance and Monitoring Mounting video cameras is cheap, but finding available human resources to observe the output is expensive. According to study of US Nat’l Institute of Justice: A person can not pay attention to more than 4 cameras. After only 20 minutes of watching and evaluating monitor screens, attention of most individuals falls below acceptable levels. Although surveillance cameras are already prevalent in banks, stores, and parking lots, video data currently is used only "after the fact". What is needed Continuous 24-hour monitoring of surveillance video to alert security officers to a burglary in progress, or to a suspicious individual loitering in the parking lot, while there is still time to prevent the crime.

3. Intelligent Surveillance A visual surveillance system combined with visual event detection methods to analyze movements, activities and high level events occurring in an environment. Event recognition module detects unusual activities, behaviors, events based on visual clues. Sends an alarm to operators when a suspicious activity is detected.

4. Visual Event Detection Applications Surveillance and Monitoring: Security (parking lots, airports, subway stations, banks, lobbies etc.) Traffic (track vehicle movements and annotate action in traffic scenarios with natural language verbs.) Commercial (understanding customer behavior in stores) Long-Term Analysis (statistics gathering for infrastructure change i.e. crowding measurement) Broadcast Video Indexing: Sports video indexing for newscasters and coaches. Interactive Environments: environment that responds to the activity of occupants Robotic Collaboration: robots that can effectively navigate their environment and interact with other people and robots. Medical: Event based analysis of cell motility Gait analysis, etc.

5. Event Types Real Time Alarms Low level alarms: Movement detectors, long term change detectors etc. Feature based spatial alarms: Specific object detection in monitored areas Behavior-related alarms: Anomalous trajectories, agitated behaviors, etc. Complex event alarms: Detection of scenarios related to multiple relational events Long Term and Large Scale Analysis Learning activity patterns of people or vehicles in a given environment over a long period of time can be used to: –retrieve events of interest –make projections –identify security holes –control the traffic or crowd –make infrastructure decisions –monitor behavior patterns in urban environments

6. Issues in High Level Video Analysis 1-Analysis: Segmentation of motion blobs (background models, shadow). Object tracking (prediction, correspondence, occlusion resolution etc.) 2-Representation: Video object representations (shape, color descriptors, geometric models). High-level event representations. 3-Access: Efficient data structures for high-dimensional feature space. Efficient and expressive query interface for query manipulation.

7. Visual Event Detection Framework Feature Extraction Motion Analysis Event Detection Object Classification -Objects -Relationships -Events Context Object, Scene & Event Libraries Events right turn cross road Constraints

8. Controlled Environment versus Far-view Outdoor Surveillance Controlled Environment (i.e. indoor) Uncontrolled Environment (i.e. far-view outdoor) IlluminationControlled generally static except light switch which cause a global change. Highly dynamic especially in cloudy days. ShadowsSmoothDarker and Sharper Object SizeLargeSmall (difficult to learn an appearance model) Perspective Distortion LowHigh Color saturationHighLow BackgroundStaticDynamic (wind etc.) Type of motionArticulatedWhole-body

9. Our Current Capabilities Moving Object Detection Moving Object Tracking Sudden Illumination Change Detection Trajectory Filtering and Discontinuity Resolution Moving Cast Shadow Detection/Elimination

10. Our Current Capabilities 1.Moving Object Detection - Using Mixture of Gaussians method or Flux tensors 2.Moving Cast Shadow Elimination 3.Sudden Illumination Change Detection 4.Moving Object Tracking – Multi-hypothesis testing using appearance and motion 5.Trajectory filtering - Temporal consistency check, spatio- temporal cluster check 6.Discontinuity resolution - Kalman filter, appearance model (color and spatial layout) } Combined photometric invariants

11. Moving Object Detection Goal: Segment moving regions from the rest of the image (background). Rationale: Provide focus of attention for later processes such as tracking, classification, event detection/recognition.

12. Background Subtraction By comparing incoming frames to a reference image (background model), regions in the incoming frame that have significantly changed are located. Feature Extraction Postprocessing BG/FG Classification ComparisonPreprocessing BG Modeling BG/FG masks Frames BG model Preprocessing -Spatial smoothing -Temporal smoothing -Color space conversions Features -Luminance -Color -Edge maps - Albedo (reflectance) image -Intrinsic images -Region statistics Comparison -Differentiation -Likelihood ratioing Postprocessing -morphological filtering -connectivity analysis -color analysis -edge analysis -shadow elimination Classification -Thresholding -Clustering

13. Problems with Basic Methods

14. Challenging Situations in Moving Object Detection 1.Moved objects: A background object that moved should not be considered part of the foreground forever after. 2.Gradual illumination changes alter the appearance of the background (time of day). 3.Sudden illumination changes alter the appearance of the background (cloud movements). 4.Periodic movement of the background: Background may fluctuate, requiring models which can represent disjoint sets of pixel values (waving trees). 5.Camouflage: A foreground objects' pixel characteristics similar to modeled background. 6.Bootstrapping: A training period absent of foreground objects is not always available. 7.Foreground aperture: When an homogeneously colored object moves, change in interior pixels can not be detected. 8.Sleeping person: When a foreground object becomes motionless it cannot be distinguished from a background. 9.Waking person: When an object initially in the background moves, both the object and the background appear to change. 10.Shadows: Foreground objects' cast shadows appear different than modeled background.

15. Background Model Color history of the specified pixelColor distribution of the specified pixel intensity Mixture of Gaussians Model The recent history of each pixel, X(1),...,X(t), is modeled by a mixture of K Gaussian distributions. Each distribution is characterized by its mean μ, variance σ 2, weight w (indicates what portion of the previous values did get assigned to this distribution).

16. Performance of Mixture of Gaussians Method 1.Moved objects √ 2.Gradual illumination changes √ 3.Sudden illumination changes X 4.Periodic movement of the background √ X 5.Camouflage X 6.Bootstrapping √ 7.Foreground aperture √ 8.Sleeping person √ 9.Waking person √ 10.Shadows X Since MoG is adaptive & multi-modal, it is robust to: Gradual illumination changes Repetitive motion of the background (such as waving trees) Slow moving objects Introduction and removal of scene objects (sleeping person & waking person problems) –when something is allowed to become part of the background, the original background color remains in the mixture until it becomes the least probable and a new color is observed.

17. Moving Object Detection using Flux Tensors Input sequence obtained from OTCBVS Benchmark Dataset Collection http://www.cse.ohio-state.edu/otcbvs-bench/ Color image sequenceThermal image sequenceMoving Objects Detected using Flux Tensors

18. merges separate objects creates “new” objects static shadow Shadow Problem

19. Shadow Detection by Combined Photometric Invariants for Improved Foreground Segmentation Moving Object Detection Identification of Darker Regions Normalized Color Comparison Reflectance Ratio Comparison Combination Post Processing Shadow Detection New Frame FGmask BG model FGmask Shadow Mask FGmask Shadow Mask

20. Combine the Masks Problems with photometric invariants: An invariant expression may not be unique to a particular material. There may be singularities and instabilities for particular values. (normalized color is not reliable around black vertex). For a robust result: Combine results from two invariants based on two different properties –Normalized color : spectral properties. –Reflectance ratio: spatial properties. –At shadow boundaries, same illuminant assumption fails. different reflectance ratios for neighbor pixels misclassification of shadow pixels as foreground dilate shadow mask.

21. Example: Intelligent Room Sequence Input Image Frame #100MOG Model #1 MOG Model #2MOG Model #3MOG Model #4

22. Shadow Masks Normalized Color MaskReflectance Ratio Mask Shadow MaskPost processed shadow mask

23. Foreground & Shadow Masks Foreground MaskPost Processed Foreground Mask Shadow MaskPost Processed Shadow Mask

24. Example: Walk-in Sequence Input Frame Walk-in #14Model 1 Model 2Model 3Model 4

25. Shadow Masks Normalized Color Masks Reflectance Ratio Mask Shadow Mask Shadow Mask Post Processed

26. Foreground & Shadow Masks Foreground Mask Foreground Mask Post Processed Shadow MaskShadow Mask Post Processed

27. Sudden Illumination Changes (Cloud Movements, Light switch etc.) Sudden illumination changes completely alter the color characteristics of the background, thus increase the deviation of background pixels from the background model in color or intensity based subtraction. Result: Drastic increase in false detection (in the worst case the whole image appears as foreground). This makes surveillance under partially cloudy days almost impossible.

28. Moving Object Tracking Object States Moving Object Detection & Feature Extraction Data Association (Correspondence) Prediction Update Context Tracking Steps: 1.Predict locations of the current set of objects of interest. 2.Match predictions to actual measurements. 3.Update object states.

29. Tracking (as a Dynamic State Estimator) Dynamic System State Estimator Measurement System state Measurements State estimate State uncertainties System Error Source Agile motion Distraction/clutter Occlusion Changes in lighting Changes in pose Shadow (Object or background models ) are often inadequate or inaccurate) Measurement Error Source Camera noise Grabber noise Compression artifacts Perspective projection States Position Appearance Color Shape Texture etc. Support map System noiseMeasurement noise

30. Our Tracking Method Detection-based Probabilistic Features Used in Data Association: –Proximity –Appearance Data Association Strategy: Multi-hypothesis testing Gating Strategies: Absolute and Relative Discontinuity Resolution: –Prediction (Kalman filter) –Appearance models Filtering: –Temporal consistency check –Spatio-temporal cluster check

31. Trajectory Filtering Some artifacts can not be totally removed by image or object level processing. These artifacts produce spurious segments.

32. Temporal Consistency Check Source of the Problem: Segments resulting from Temporarily fragmented parts of an object Un-eliminated cast shadows Effect: Short segments that split from or merge to a longer segment. Proposed Solution: Pruning short split or merge segments by temporal consistency check. Elimination of short disconnected segments are delayed until after discontinuity resolution.

33. Spatio-Temporal Cluster Check Source of the Problem: Repetitive motion of the background (i.e. moving branches or their cast shadows). Spectral reflections (i.e. reflections from car windshields). Effect: Temporally consistent and spatially clustered trajectories. Proposed Solution: Average Displacement to Length Ratio (ADLR) Diagonal to Length Ratio (DLR)

34. Discontinuity Resolution Discontinuities occur especially in low resolution outdoor sequences. Source of the problem: Temporarily undetected objects due to –Low contrast –Partial or total occlusions Incorrect pruning in data association due to significant change in appearance or size caused by –Partial occlusion –Fragmentation

35. Discontinuity Resolution 1.Define source and sink locations where the objects are expected to appear and disappear. 2.Identify –Seg dis : Segments disappearing unexpectedly (at a non-sink location) -> possible start of a discontinuity. –Seg app : Segments appearing unexpectedly (at a non-source location) -> possible end of a discontinuity. 3.Identify possible matches based on time constraint. 4.Use Kalman filter to predict future positions of disappearing and past positions of appearing segments. 5.Check direction and position consistencies on –Disappearing segment –Appearing segment –Joining segment 6.Check Color similarity. 7.Multiple possible matches for a single disappearing segment-> select appearing segment starting earliest. 8.Multiple possible matches for a single appearing segment-> select disappearing segment ending latest. 9.Match-> appearing segment inherits disappearing segment’s label and propagates this new label to its children.

36. Challenges in Tracking for Visual Event Detection Shadows -false detections, shape distortions, merges Sudden illumination changes (e.g. due to cloud movements) -difficulty in object detection especially in partly cloudy days Glare from specular surfaces (e.g. car windshields) -spurious detections and trajectory segments Perspective distortion (objects far away from the camera look smaller and appear to move slower) -difficulty in filtering false detections Occlusion -discontinuities in trajectories Poor video quality (low resolution, low color saturation) -difficulty in moving object detection -difficulty in appearance modeling

37. Some Experimental Results-1 a) All segments b) Pruned segments c) Predictions d) After discontinuity resolution

38. Some Experimental Results-2 a) All segmentsb) Pruned segments c) Predictionsd) After occlusion handling UPS

39. Some Experimental Results-3 a) All segmentsb) Pruned segments c) Predictions d) After discontinuity resolution

40. Potential Collaborations in Visual Event Detection New moving object detection methods –Flux tensor (especially in the presence of global motion, clutter and illumination changes) –Weather (i.e. snow, rain, wind) Trajectory analysis –Trajectory validation –Feature extraction –Trajectory annotation Extraction of primitive events based on –Trajectory properties –Trajectory to trajectory interactions –Agent types Complex event detection/recognition through temporal combination of primitive events –Hierarchical approach Low-level : probabilistic methods High-level : structural methods Incorporation of learning to event modeling and recognition. Video event mining

41. Questions?