1. Project 35 Visual Surveillance of Urban Scenes

2. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Principal Investigators David Clausi, Waterloo Geoffrey Edwards, Laval James Elder, York (Project Leader) Frank Ferrie, McGill (Deputy Leader) James Little, UBC

3. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Partners Honeywell (Jeremy Wilson) CAE (Ronald Kruk) Aimetis (Mike Janzen)

4. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Participants Postdoctoral Fellows Francisco J. Estrada (York) Bruce Yang (Waterloo) Students Eyhab Al-Masri (Waterloo) Kurtis McBride (Waterloo) Natalie Nabbout (Waterloo) Isabelle Begin (McGill) Albert Law (McGill) Prasun Lala (McGill) John Harrison (McGill) Antoine Noel de Tilly (Laval) Samir Fertas (Laval) Michael Yurick (UBC) Wei-Lwun Lu (UBC) Patrick Denis (York)

5. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Goals Visual surveillance of urban scenes can potentially be used to enhance human safety and security, to detect emergency events, and to respond appropriately to these events. Our project investigates the development of intelligent systems for detecting, identifying, tracking and modeling dynamic events in an urban scene, as well as automatic methods for inferring the three- dimensional static or slowly-changing context in which these events take place.

6. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Results Here we demonstrate new results in the automatic estimation of 3D context and automatic tracking of human traffic from urban surveillance video. The CAE S-Mission real-time distributed computing environment is used as a substrate to integrate these intelligent algorithms into a comprehensive urban awareness network.

7. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Proprietary CAE Inc 2007 dispatcher logic HLA logs other types of logs historic data CAE STRIVE ARCH.

8. 3D Urban Awareness from Single-View Surveillance Video

9. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES 3D Urban Awareness 3D scene context (e.g., ground plane information) is crucial for the accurate identification and tracking of human and vehicular traffic in urban scenes. 3D scene context is also important for human interpretation of urban surveillance data Limited static 3D scene context can be estimated manually, but this is time-consuming, and cannot be adapted to slowly-changing scenes.

10. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES

11. Ultimate Goal Our ultimate goal is to automate this process!

12. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Immediate Goal Automatic estimation of the three vanishing points corresponding to the “Manhattan directions”.

13. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Manhattan Frame Geometry An edge is aligned to a vanishing point if the interpretation plane normal is orthogonal to the vanishing point vector in the Gaussian Sphere (i.e. dot product is 0)

14. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Mixture Model Each edge E ij in the image is generated by one of four possible kinds of scene structure: –m 1-3 : a line in one of the three Manhattan directions –m 4 : non-Manhattan structure The observable properties of each edge E ij are: –position –angle The likelihoods of these observations are co-determined by: –The causal process (m 1-4 ) –The rotation Ψ of the Manhattan frame relative to the camera mimi mimi mimi mimi E 11 E 12 E 22 E 21 Ψ Image

15. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Mixture Model Our goal is to estimate the Manhattan frame Ψ from the observable data E ij. mimi mimi mimi mimi E 11 E 12 E 22 E 21 Ψ Image

16. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES E-M Algorithm E Step –Given an estimate of the Manhattan coordinate frame, calculate the mixture probabilities for each edge m1m1

17. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES E-M Algorithm E Step –Given an estimate of the Manhattan coordinate frame, calculate the mixture probabilities for each edge m2m2

18. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES E-M Algorithm E Step –Given an estimate of the Manhattan coordinate frame, calculate the mixture probabilities for each edge m3m3

19. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES E-M Algorithm E Step –Given an estimate of the Manhattan coordinate frame, calculate the mixture probabilities for each edge m4m4

20. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES E-M Algorithm M Step –Given estimates of the mixture probabilities for each edge, update our estimate of the Manhattan coordinate frame

21. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Results

22. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Results Convergence of the E-M algorithm for example image Test Image

23. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Results Example: lines through top 10 edges in each Manhattan direction

24. Tracking Human Activity

25. Single-Camera Tracking

26. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Tracking Using Only Colour / Grey Scale Tracking using only grey scale or colour features can lead to errors

27. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Tracking Using Dynamic Information Incorporating dynamic information enables successful tracking

28. Tracking over Multi-Camera Network

29. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Goal Integrate tracking of human activity from multiple cameras into world-centred activity map

30. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Input left and right sequences

31. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Independent tracking Each person tracked independently in each camera using Boosted Particle Filters. –Background subtraction identifies possible detections of people which are then tracked with a particle filter using brightness histograms as the observation model. Tracks are projected via a homography to the street map, and then Kalman filtered independently based on the error model.

32. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Independent tracks

33. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Integration Tracks are averaged to approximate joint estimation of composite errors

34. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Merged trajectories

35. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Future Work Integrated multi-camera background subtraction Integrated particle filter in world coordinates using joint observation model over all sensors in network.

36. Tracking in Dynamic Background Settings

37. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Foreground Extraction and Tracking in Dynamic Background Settings Extracting objects from dynamic backgrounds is challenging Numerous applications: –Human Surveillance –Customer Counting –Human Safety –Event Detection In this example, the problem is to extract people from surveillance video as they enter a store through a dynamic sliding door

38. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Methodology Overview Video sequences are pre-processed and corner feature points are extracted Corners are tracked to obtain trajectories of the moving background Background trajectories are learned and a classifier is formed Trajectories of all moving objects in the test image sequences are classified based on learned model into either background or foreground trajectories Foreground Trajectories are kept in image sequence and the object corresponding to those trajectories is tagged as foreground

39. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Demo 1: Successful Tracking and Classification This demo illustrates a case of successful tracking and classification of an entering person.This demo illustrates a case of successful tracking and classification of an entering person. The person is classified into foreground based on the extracted tr.The person is classified into foreground based on the extracted trajectories.

40. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Demo 2: Failed Tracking but Successful Classification Demo 2 shows a case when the tracker loses track of the person after a few frames However, the classification is still correct since only a small number of frames are required to identify the trajectory.

41. Recognizing Actions using the Boosted Particle Filter

42. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Motivation Frame 682Frame 814 Input Output

43. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES update the SPPCA template updater System Diagram Extracted image patches predict new templates

44. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES HSV Color Histogram The HSV color histogram is composed of: –2D histogram of Hue and Saturation –1D histogram of Value + Hue Saturation 2D histogram Value 1D histogram

45. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES The HOG descriptor SIFT descriptor Image gradients

46. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES ? Template Updating: Motivation Tracking: search for the location in the image whose image patch is similar to a reference image patch – the template. Template Updating: Templates should be updated because the players change their pose. ? ? ? Frame 677Frame 687

47. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Template Updating: Operations Offline –Learning: Learn the template model from training data Online: –Prediction: Predict the new template used in the next frame –Updating: Update the template model using the current observation

48. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES SPPCA Template Updater New templates Update the SPPCA template updater Predict new templates

49. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Graphical Model of SPPCA observation (continuous) coordinate on the Eigen space (continuous) switch to select an Eigen space (discrete)

50. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Action Recognizer Input: a sequence of image patches Output: action labels Action Recognizer skating down skating left skating right skating up

51. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Summary: –Features: The HOG descriptor –Classifier: The SMLR classifier –Weights: weights learned by MAP estimation with a sparsity-promoted Laplacian prior –Basis functions : motion similarity between the testing and training data Action Recognizer

52. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Action Recognizer: Framework Compute the frame-to-frame similarity Convolve the frame similarity with the weighting matrix SMLR classifier Action labels HOG descriptors

53. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Tracking & Action Recognition Frame 97Frame 116Frame 682Frame 710Frame 773Frame 814

54. PROJECT 35: VISUAL SURVEILLANCE OF URBAN SCENES Vehicle Tracking