Kien A. Hua Division of Computer Science University of Central Florida

Live Video Computing (LVC) Data Systems Group, Department of Electrical Engineering and Computer Science 2 Conventional Computing Process data captured on disks Live Video Computing Process live video feeds

Platform for Software Development Data Systems Group, Department of Electrical Engineering and Computer Science 3 Current Approach: Customized software development Proposal: A Live Video Database Management System (LVDBMS) Motivation: A general platform enables rapid development of LVC applications much like database applications are developed today A special class of storage

We focus on Video Surveillance Data Systems Group, Department of Electrical Engineering and Computer Science 4

Video Surveillance Challenge  Problem: who is watching them Impractical to scale number of human operators as the number of cameras increases Issues include fatigue, taking breaks, distraction, …  Traditional video processing solutions are vertical applications e.g., a traffic monitoring application cannot be used to monitor patients in a hospital Data Systems Group, Department of Electrical Engineering and Computer Science 5 Cost effective to deploy many cameras Numerous live video streams

6 Domain-Specific Systems IP 1 SP 1 IP 2IP 3 SP 2SP 3 IP: Image Processor SP: Surveillance Processor Application specific, limited or no programming support A specific set of known cameras Observation Sharing similar underlying software Observation Sharing similar underlying software Systems customized for different applications

Live video streams as database content 7 LVDBMS Approach IP 1 SP 1 IP 2IP 3 SP 2SP 3 IP Different user groups share the cameras for different purposes IP: Image ProcessorSP: Surveillance Processor Query interface LVDBMS Programmable, not application specific

Cross-Camera Object Tracking  Object tracking is essential to a LVDBMS  Standard face recognition techniques Computationally expensive and may not be suitable for real-time applications Video frames may not have enough resolution for face analysis  Observation: We typically want to recognize reappearance of objects within minutes or hours Comparing overall appearance of objects is a much easier task Data Systems Group, Department of Electrical Engineering and Computer Science 8

Informatics-based Approach Data Systems Group, Department of Electrical Engineering and Computer Science 9 LOCATION ALOCATION B Insert at time t 1 Live Database Jane moves from A to B  A new object is used to retrieve similar objects recently captured by the database system  By logging the sequence in which the different cameras detect the same object we can track its trajectory Jane = Retrieve at time t 2

Object Representation Data Systems Group, Department of Electrical Engineering and Computer Science 10 Each snapshot of an object is represented as a visual feature vector (i.e., a point in the multidimensional space) Each object is represented as a point set allowing multiple views/orientations to contribute to representation Multifaceted Object Model

Object Feature Extraction  Object detected and segmented in stream  Feature vectors (FVs) extracted from sample frames  Object represented as bag of feature vectors over time in stream Data Systems Group, Department of Electrical Engineering and Computer Science 11 Video frames FV A point set Sample frame Window for point set extraction

Object Comparison Let X and X’ be two bags (point sets) Data Systems Group, Department of Electrical Engineering and Computer Science 12 Minimum sums of pairwise distances Squared distance

m: Matching Cardinality m = 1m = 2m = 3 A point set can be viewed as a polygon k is the cardinality of the bags. m ≤ k is the matching cardinality. It controls the number of facets considered by the system.

Dual-Layer Computation Framework Data Systems Group, Department of Electrical Engineering and Computer Science 14 Motion Imagery Processor Stream Processor Query results Process spatial operations over abstract objects to convert real-time imagery information into bit streams Process temporal operations over bit streams to detect events, i.e., correlate objects across video streams to detect complex events Camera as a special class of storage system A live video database management system

Distributed Live Video Database 15 Hardware Environment Query Results

16 Event Model – Spatial Event  Spatial event E s (t 1 ) = disjoint(O i (t 1 ), O j (t 1 ))  Spatial event stream Stream of spatial events on two objects E s = {T, T, T, T, T, T, F, F, F} o E s (t 1 ) = TE s (t 2 ) = TE s (t 3 ) = T E s (t 4 ) = T E s (t 5 ) = T E s (t 7 ) = F E s (t 6 ) = T E s (t 8 ) = F E s (t 9 ) = F Snapshot of an object

17 Event Model – Composite Event E c → E s | T o (E c1, E c2 ) | L o (E c1, E c2 ) Spatial Event e.g., Overlap(O 1, O 2 ) Temporal Operator e.g., before(E s1, E s2 ) Logical Operator e.g., E s1 and E s2 Composite Event

18 Query language ACTION ON EVENT E.g., archive the video for the last 15 minutes An event composition expression Processing window, e.g., last 15 minutes

19 Query Translation during before overlap o1 o2 overlap o3 o4 contains o1 o2 appears o3 (overlap(v1.o1,v1.o2) and overlap(v1.o3, v1.04)) before (contains(v2.o1, v2.o2) during appear(v2.03)) last 1000 Parse the expression to construct the execution tree Logical operator Temporal operator Spatial operator

20 Query Processing: Dual Layers Video Processor Stream Processor Results Convert video information into bit streams Process the bit streams to detect complex events

21 Distributed Processing Query (overlap(v1.o1, v1.o2) and overlap(v1.o3, v1.o4)) before (contains(v2.o1, v2.o2) during appears(v2.o3)) 1000 Query translation Sub-query results Further processing Results Sub-queries

Privacy Filters  Privacy filters can be defined using a privacy specification language  Privacy filters can be used to implement privacy policies – when and under what circumstances the identity of an individual may be observed or recorded  Privacy filters can be declared at the different levels of the system hierarchy, e.g., at the individual camera level or associated with specific queries or users

Privacy Views vs. Relational Views 23 IDNameAddress 12Fred12 East Bullard IDName 12Fred Physical table Table1 Name Fred View 1 defined from Table1 View2 defined from View1 F2F2 Terminal 1 F1F1 Terminal 2Terminal 3 User Without Privacy Filter User Privacy Filter Query Privacy Filter Video Stream F1F1 Privacy Filter 1 F2F2 Privacy Filter 2

Privacy Filter Examples 24 Video observed by all users Video observed by User 2 Removed from all consumers of this camera Plaque redacted from view of all users All moving objects hidden from User 2

Privacy Certification Having a standardized privacy specification language is an important step toward privacy certification A privacy specification language allows for a formal way to design, verify, test, and deploy privacy policies

Let me know when someone uses my laptop computers at home User Interface

LVDBMS: Summary  Object Tracking The multifaceted object model is more indicative of the objects The informatics-based approach to object tracking is effective. It does not require assumptions such as known topography or boundaries (walls, roads, etc.)  LVDBMS provides a general platform for live video computing A variety of video surveillance applications can be quickly developed atop the LVDBMS much like many applications are developed using a DBMS today  Current Limitation Limited object recognition capability Data Systems Group, Department of Electrical Engineering and Computer Science 27