Automated video surveillance: challenges and solutions. ACE Surveillance (Annotated Critical Evidence) case study. Dmitry Gorodnichy and Tony Mungham Laboratory & Scientific Services Directorate Canada Border Services Agency

2 Outline Problems with status-quo Video Surveillance Real-time and archival problems Operational considerations Next generation solution - Video Analytics based “Motion detection” myth and problem “Object detection” as example of real intelligence ACE Surveillance – first fully-functional object- detection-based prototype Year long tests with different levels of complexity What that means for future of Video Surveillance Conclusions

3 Role of Video Technology (VT) In the context of enhancing security, Video Technology (VT) is one of the most demanded technologies of the 21st century It is publicly acceptable It provides rich in content data Multi-million funding in Canada and worldwide: CBSA Port Runner project invested 10s of Millions in CCTV upgrades Transport Canada opens $35M of funding towards procurement of CCTV

4 VT at CBSA CBSA is a major user of CCTV systems at POEs Most major CCTV installations start to leverage VT Current task: to lead applied R&D to push VT to help CBSA apply S&T innovative approaches to border management: Event detection and notification to provide effective response to events Traffic trends analysis to assist with border management Video storage management to manage the cost of storage and meet obligations under the privacy act Data integration/fusion of contextualised video information

5 Problem with status quo use of CCTV surveillance Modes of operation: 1. Active - personnel watch video at all times 2. Passive - in conjunction with other duties 3. Archival - for post-event analysis Current systems and protocols are not efficient in either mode! Problem in real-time modes: an event may easily pass unnoticed. due to false or simultaneous alarms, lack of time needed to rewind and analyse all video streams.

6 Problems in Archival mode: Due to temporal nature of data: 1. Storage space consumption problem Typical assignment: 2-16 cameras, 7 or 30 days of recording, 2-10 Mb / min.  1.5 GB per day per camera / GB total ! 2. Data management and retrieval problem London bombing video backtracking experience: “Manual browsing of millions of hours of digitized video from thousands of cameras proved impossible within time- sensed period” [by the Scotland Yard trying to back-track the suspects]

7 Operational considerations Lots of CCTV infrastructure: Many local initiatives, not coordinated Most video technology decisions are influenced by vendors - short-term solutions Over 30 different video systems within the same dept. (at RCMP) A national program with proper benchmark-based planning and evaluation of VT is required Leveraging advances recently made in S&T Technical standards for capturing /saving video data. Policies in when, where and how VT should be used.

8 Video Technology today Video Analytics (Video Recognition) Analog 20 th century 21 st century First video recording Digital Wireless, Network Connected (IP)

9 Next generation Video Technology Is Video Analytics based also identified as: Video Recognition, Intelligent Video, Smart Video / Smart Camera Video Analysis & Content Extraction Perceptual Vision is not much about capturing better data (better lenses, grabbers, coders, transmitters) but about understanding captured data (better theory)

10 Status-quo “video intelligence” Transport Canada CCTV Reference Manual for Security Application. Australian Government National code of practice for CCTV applications in urban transport USA Government :recommended security Guidelines for Airport Planning, Design and Construction. …. refer to “Motion-based” capture as Intelligent Surveillance Technology, and make their recommendations based on thereon.

11 “Motion-detection” is not intelligent! Term “Motion-based” is coined to make people believe that video recognition is happening, which is not! It’s actually illumination-change-based, as it uses simple point brightness comparison: Which often happens not because of motion! Changing light / weather (esp. in 24/7 monitoring) Against sun/light, out of focus, blurred, thru glass Reflections, diffraction, optical interferences Image transmission, compression losses

12 “Object-detection” is intelligent … … but few can do it, since necessary advances in video recognition theory became possible only recently (>2002). In 2002 National Research Council of Canada (NRC) starts developing Video Recognition Systems to leverage its scientific Video Recognition expertise for the industry. In 2005, it develops ACE Surveillance: an object-detection-based Automated surveillanCE prototype capable of automatically extracting Annotated Critical Evidence from live video. NRC becomes also the organizer of the first Canadian academic workshops dedicated to Video Processing for Security (since 2004)

13 What is ACE Surveillance? A Windows software that performs real-time video analytics by integrating best object detection and tracking algorithms. Replaces video clips with annotated still images: Compresses 1 Gb of video into 2 Mb of easy to browse and analyze still images ACE Surveillance output: A 7-hour activity from day to night (17: :00) is summarized into 2 minutes (600Kb) of Annotated Critical Evidence snapshots. Note illumination changes! - Watch tree shadows and sun light.

14 ACE Surveillance architecture Works with ordinary USB cameras or CCTV cameras with USB video converters. Adds on top of existing infrastructure using an ordinary desktop computer. Video clips (Tb)... Archival mode of operation Real-time mode of operation Last captured CES ALARM! C.E.S. (Gb) ACE Capture ACE Browser

15 Adds on top of existing infrastructure

16 Status quo “Motion-based” capture (Courtesy: NRC-IIT Video Recognition Systems project) 1. Many captured snapshots are useless: either noise or redundant 2. Without visual annotation, motion information is lost. 3. Hourly distribution of snapshots is not useful

17 ACE Surveillance “Object-based” capture (Courtesy: NRC Video Recognition Systems project) 1. Each captured shot is useful. 2. Object location and velocity shown augmentent. 3. Hourly distribution of shots is indicative of what happened in each hour, provides good summarization of activities over long period of time.

18 ACE Surveillance testing benchmarks Tested in different levels of complexity: lighting conditions, object motion patterns, camera location environmental constraints. most difficult - outdoors in unconstrained environments with little or no object motion consistency (as around a private house in a regular neighbourhood). most easy - in controlled indoor environment where minimal direct sunlight is present and where all objects are of approximately the same size and exhibit similar motion pattern (as at access gate inside the business building).

VT within CBSA19 Camera / setup Annotated CES ACE daily summarization Outdoor, wireless, eye-level Outdoor, webcam, overviewIndoor with sunlight, CCTVIndoor w/o sunlight, CCTV

20 Enables efficient detection of abnormal activities Delivery EntryBack Door Entry On week-day On week-end More than usual

21 ACE Surveillance results In real-time mode: alarm sounds & last captured evidence (time-stamped) is shown. In archival mode: “Zoom on the evidence” browsing of captured evidences – zoom on a day, on hour, then on event - point and click (for high res as needed) Made Commissioners much more aware of activities.

22 Conclusions Affordable automated (intelligent) video surveillance (AVS) is possible! To replace traditional DVR OR to supplement them: DVR for 1 month + AVS for 1 year However: Requires extra training from security officers. Requires new protocols to handle automatically extracted evidence. - From forensic prospective, data that are not original and have been processed by a computer can not be considered as evidence. Requires new privacy policies. - Surveillance data are normally not kept for a long period of time (<1 month), due to their size. AVS allows to store on local machine many months (even years) of evidence data.

23 ACE surveillance case study outcome ACE Surveillance (which is developed by a research lab) provides a reference standard against which can be measured solutions coming from industry. It deals with common misconceptions related to deploying intelligent video surveillance systems (IVS): “motion detection” myth vs real object detection and tracking. The “one-fit-all” myth. - Extra video analytics expertise is required to set and operate IVS. better video data (better resolution or compression) do not imply better video intelligence. - ACE Surveillance is shown to work with regular TV quality data (320 by 240 pixels). However better quality of video image is needed for forensic purposes as evidence Due to closing of the project by NRC, CBSA takes lead on it.