Video Analysis Tool Box for Digital Video Forensics By Susinda Perera Department of Computer Science and Engineering, University of Moratuwa, Supervised by Dr. Chathura De Silva PhD (NUS-Singapore), MEng (NTU-Singapore), BSc Eng.(Hons) (Moratuwa) Senior Lecturer Department of Computer Science and Engineering, University of Moratuwa,

Problem Statement Can we trust digital videos? – Are they real, computer generated or tampered Extract some wanted Information from video – Difficult due to unclearness of video

Can we trust digital videos? Figure 1 ‑ 1 : A still from controversial video aired on Channel 4

Unclear Videos

Solution Video Stream analysis tool Video Enhancement tool

Video Stream Analyzing Tools Elecard StreamEye Tektronix MPEG Software Tools MPEG-2 Transport Stream packet analyser TSReader MTS4EA Elementary Stream Analyzer

Screenshots from Elecard StreamEye

Some Features Of Video Stream Analyzing Tools Navigation and display of media stream picture-by-picture (I, P, B). Display of the current frame. Display of the time, type, size and number of a current frame in a stream, decoding order and offset from the file beginning. Display of the bit rate (declared in the sequence header) and a calculated bit rate. Display of detailed information about macroblocks in MPEG-1 (ISO/IEC ), MPEG 2 (ISO/IEC ), MPEG-4 (ISO/IEC ) and AVC/H.264 (ISO/IEC ) video streams. Information about motion vectors Frame-accurate positioning. Display of the stream and gathering of statistics relating to the entire file.

Video Enhancement Software Cognitech Ocean Systems dTective Salient Stills VideoFOCUS StarWitness Avid Technology, Inc. Intergraph Video Analyst TREC, Inc. Forevid MotionDSP Ikena Amped FIVE Kinesense Cellforensics (Video Recovery from Mobile Device)

Some Features Of Video Enhancement Tools Video Stabilization Denoising Deblur Filters Detection Filters Enhancement Histogram Editor Segmentation Tracking Transform Zoom Velocity Reconstruction

Literature Review Stream analysis tools – Decoder Libraries/Source Codes Video enhancement tools – Algorithms, Research papers

Decoder Libraries/Source Codes libmpeg2 - a free MPEG-2 video stream decoder – MPEG-2 Video Codec - by MPEG Software Simulation Group (MSSG) – FFmpeg - a complete, cross-platform solution to stream audio and video. Includes libavcodec - the leading audio/video codec library – MPEG2Event - C# library intended to facilitate rapid prototyping of MPEG-2 analysis tools. – ml ml Berkeley mpeg_play source code –

MPEG2Event Issues – Library accepts only the elementary video streams – Needs MPEG Demultiplexer DirectShow Filters – Creates a Large row file – Needs lisence MPEG Demuxers on internet – Bit stream not compatible with library – MPEG standard(ISO) – Library is vent based – very slow operation

Video Enhancement Algorithms/Techniques – Motion estimation – Correlation Matching – Line Segment Matching – Motion Segmentation – Object tracking – Median and Average Frames – Total Variation Denoise – + many more……….

Video Stabilization Removing annoying shaky motion from videos helpful in identifying people, license plates, etc. from low-quality video cameras Three aspects – Inter frame motion estimation – Motion smoothing and compensation – Filling up the missing image areas.

Video Stabilization

Main references Full-frame Video Stabilization, Yasuyuki Matsushita, Eyal Ofek Xiaoou Tang, Heung-Yeung Shum Video Stabilization Using Scale-Invariant Features Rong Hu, Rongjie Shi, I-fan Shen, Wenbin Chen Video Stabilization and Enhancement, Hany Farid and Jeffrey B. Woodward, Department of Computer Science, Dartmouth College

Motion Estimation Computing inter frame motion – Use of object recognition – Invariant Feature Transform(SIFT) features – Minimizing quadratic error function with a proposed model

Motion Model f(x, y, t) = frame at time t f(x, y, t − 1) = frame at time t -1 T = affine transform – f(x, y, t) = T * f(x, y, t − 1) – The three papers mentioned above use different mecanisms to find the transform matrix parametrs

Motion Smoothing A stabilized motion path is obtained by removing undesired motion fluctuation. Assumed that the intentional motion in the video is usually slow and smooth Uses Gaussian kernel in most literatures – Applies Gaussian kernel to neighboring N frames Gaussian kernel + curve fitting methods

Motion Smoothing Let N t = {j|t-k<=j<=t+k} be the neighboring frames And I t is the frame at the origin Calculate the position of each neighboring frame I s, relative to frame I t using transform matrixes defined above ( lets say T s t ) Find the correcting transformation S from the original frame I t to the motion-compensated frame I’ t according to Where G is a Gaussian kernel

Motion Smoothing The global transformation chain T defined over the original video frames I i, and the transformation from the original path to the smoothed path S.

Filling up missing image areas To be decide ?? Some techniques used in research literature – Motion Inpainting the local motion data in the known image areas is propagated into the missing image areas. The propagation starts at pixels on the boundary of the missing image area. Using motion values of neighboring known pixels Motion values on the boundary are defined and the boundary gradually advances into the missing area until it is completely filled – Use of dynamic programming

Noise Removal (Denoising Filters) Nonmotion compensated spatiotemporal Motion compensated spatiotemporal Nonmotion compensated temporal Motion compensated temporal filters What is first? motion compensation or denoising?

Noise Removal- Research Literature Denoising image sequence does not require motion estimation Denoising with motion estimation Adaptive weighted averaging (AWA) filter

Project output Study of techniques/algorithms used in commercial video analysis tools Literature review of video enhancing techniques Video analysis tool box – Stream analysis tool – Video enhancement tool with some number of features (not yet decided) Tampered video detection approach based on stream analyzing

References [8] The H.264/AVC Video Coding Standard. [Online]. utshell.pdf utshell.pdf [9] MPEG-2 White Paper. [Online]. nal/TopTabItems/products/dc1000/WhitePapers/DC10 00-DVD1000MPEG2whitepaper.pdf nal/TopTabItems/products/dc1000/WhitePapers/DC10 00-DVD1000MPEG2whitepaper.pdf [10] MPEG encoding basics. [Online]. matters.net/docs/resources/Digital%20Files/MPEG/MP EG%20Encoding%20Basics.pdf matters.net/docs/resources/Digital%20Files/MPEG/MP EG%20Encoding%20Basics.pdf

[11] Wikipedia. [Online]. [14] R. P. Kleihorst, S. Efsratiadis, A. K. Katsaggelos, and R. L. Lagendijk J. C. Brailean, "Noise reduction filters for dynamic image sequences: A review," Proceedings of the IEEE, vol. 83, pp , [15] Rongjie Shi, I-fan Shen, Wenbin Chen Rong Hu, "Video Stabilization Using Scale-Invariant Features," Fudan University, Handan Road 220, Shanghai, China,. [16] Eyal Ofek, Xiaoou Tang, Heung-Yeung Shum Yasuyuki Matsushita, "Full-frame Video Stabilization," Microsoft Research Asia, Haidian District, Beijing , P. R. China, [17] Jeffrey B. Woodward Hany Farid, "Video Stabilization and Enhancement," Department of Computer Science,. [Online]. [18] Janusz Konrad, William C. Karl Andrew Litvin, "Probabilistic video stabilization using Kalman filtering and mosaicking," ECE department, Boston University, 8 St. Mary’s Street, Boston

[19] P. Anandan, K.J. Hanna, and R. Hingorani J.R. Bergen, "Hierarchical model- based motion estimation," Proc. of 2nd European Conf. on Computer Vision, pp , [20] B. Coll, J. M. Morel A. Buades, "Denoising image sequences does not require motion estimation,“ [21] William T. Freeman Ce Liu, "A High-Quality Video Denoising Algorithm based on Motion Estimation,". [22] M.I.Sezan, A.M. Tekalp M.K. Ozkan, "Adaptive motion compensated filtering of noisy image sequences," IEEE Trans, circuits, vol. CSVT-3, pp , Aug [23] B. Coll, J. M. Morel A. Buades, "A non-local algorithm for image denoising," IEEE International Conference on Computer Vision and Pattern Recognition, [24] Sergei V. Fogel, "The estimation of velocity vector fields from time-varying image sequences," CVGIP: Image Understanding, vol. Volume 53, no. Issue 3, pp , May 1991.