1 Privacy Protected Video Surveillance Sen-ching Samson Cheung Center for Visualization & Virtual Environments Department of Electrical & Computer Engineering University of Kentucky

2 Acknowledgements Graduate Students Vijay Venkatesh Mahalingam Jian Zhao Jithendra K. Paruchuri Research Support Department of Homeland Security Oak Rridge Associated Universities

3 What is privacy? To develop human excellence without interference [Aristotle’s Politics 350 B.C.] Control over information about oneself [Warren and Brandeis 1890] …the right most valued by all civilized men — the right to be let alone. - U.S. Supreme Court Justice Louis Brandeis, 1890 …the right most valued by all civilized men — the right to be let alone. - U.S. Supreme Court Justice Louis Brandeis, 1890

4 Today’s privacy concerns Electronic Voting E-commerce Medical Records Financial Records Cyber Activities

5 Smart video surveillance Biometric theft Multimedia processing RFID tracking Tomorrow’s privacy concerns

6 Privacy Protection Technology Technologies that aim at protecting personal privacy without compromising the “legitimate” use of information. Main PPT include the followings: Encryption Tools Platform for Privacy Preferences (P3P) Automated Privacy Audit Anonymizer Privacy Protected Data mining Is privacy protection of multimedia any different?

7 Challenges from Multimedia What to protect? Identify semantic objects for protection How to protect it? Reliable protection without losing perceptual utility How to control it? Flexible control and secure authentication of privacy data

8 Talk Overview Video Surveillance Subject Identification Optimal Camera Network Video Obfuscation Privacy Data Management Portable AV devices Evaluation of audio privacy protections Secure Distributed Processing

9 Overview Object Identification and Tacking Obfuscation Secure Data Hiding Surveillance Video Database Subject Identification Module Secure Camera System Privacy Data Management System

10 Talk Overview Video Surveillance Subject Identification Optimal Camera Network Video Obfuscation Privacy Data Management

11 Video Obfuscation Original Pixelation/ Blurring Black Out In-painted

12 Challenges of Video Inpainting

13 Dynamic Object In-painting Basic idea: Using object template extracted form other time instant to complete a conceptually consistent sequence. Steps: 1. Similarity based on optimal alignment 2. Motion continuity 3. Positioning of templates ?

14 Motion Continuity ??

15 Object-based Video In-painting Better motion in-painting by better registration and task separation Capable to in-paint partially and completely occluded objects Improved computational performance (Matlab) Number of frames with complete occlusion Number of frames with partial occlusion

16 Public-domain Sequences

17 Complex Sequences

18 Talk Overview Video Surveillance Subject Identification Optimal Camera Network Video Obfuscation Privacy Data Management

19 Privacy Data Management Subject A Subject B Subject C Producer Client A Client B Client C Privacy Protection Key question: How does a client know which subject to ask?

20 Three-agent architecture Mediator Agent PK M,SK M Subject Agent PK U,SK U Client Agent PK C,SK C RSA(TOC; PK m ) Step 2 RSA(K; PK U ) TOC: U Step 3 RSA(K; PK U ) Step 4 RSA(K; PK C ) Step 5 RSA(K;PK C ) Step 6 Step 7 AES(V u ; K) Step 1 RSA(K; PK U )U RSA(TOC; PK m ) AES(V u ; K)

21 Keeping sensitive information MediumMethodPro Con Separate File Encryption + Cryptographic signature Standard Technology Storage efficiency Pervious to attacker Difficult to distribute with the modified video Separate authentication for modified video Meta-dataEncryption + Cryptographic signature Standard Technology Storage efficiency Less pervious to attacker Depend on format Data hiding Encrypted watermark Impervious to attacker Inseparable from data Joint authentication May need more storage May affect visual quality

22 Data Hiding for Privacy Data Preservation

23 Data Hiding Data hiding/Stenography/Watermarking Active research in the past fifteen years Typical applications include authentication, copy detection, monitoring Challenges in our application: Picture-in-picture: large embedding capacity Compatibility with existing compression scheme Minimal visual distortion

24 Optimal Data Hiding Psycho-visual Modeling Block-based Rate-Distortion Calculation Discrete Optimization Solve constrained optimization Combined rate- distortion cost C(x) Combined rate- distortion cost C(x) # embedded bits

25 Proposed Data Hiding Motion Compensation DCT Entropy Coding DCT Domain Frequency, contrast and luminance masking [Watson] H.263 Encrypted foreground video bit-stream DCT Perceptual Mask Parity Embedding R-D Optimization Positions of the “optimal’ DCT coeff for embedding DCT(i,j) = watermark_bit+ 2*round(DCT(i,j)/2) Privacy protected video Last decoded frame J. Paruchuri & S.-C. Cheung “Rate- Distortion Optimized Data Hiding for Privacy Protection” submitted to ISCAS 2008

26 R-D framework Target cost function: R i = Increase in Bandwidth of Block i D i = Perceptual Distortion in Block i δ = Relative Weight Greedy embedding of P data bits in Block i: Lagrangian optimization: determine the optimal P i and λ to embed the target number of data bits:

27 Examples 1/2 119kbps No data Distortion 637 kbps 81 kbps data Rate & Distortion 562 kbps 81 kbps data Rate only 370 kbps 81 kbps data

28 Examples 2/ kbps No data Distortion 743 kbps 81 kbps data Rate & Distortion 678 kbps 81 kbps data Rate only 610 kbps 81 kbps data

29 Conclusions Privacy Protecting Video Surveillance Visual Tagging for subject identification Optimized camera network for visual tagging In-painting for video obfuscation Privacy Data Management R-D optimized watermarking Current Research Video In-painting in crowded environment Performance Evaluation for PPT Secure Reversible Modification Audio Privacy Protection Signal processing in encrypted domain